How our crawler works
This directory is built by reading affiliate pages that companies publish on their own sites. The crawler that does it identifies itself honestly on every request, and this page is what that identification points at.
How to recognise it
Every request we make carries this User-Agent:
LinkedAffiliateBot/0.1 (+https://linkedaffiliate.com/crawler)We never pretend to be a browser, and we never try to get around a block. If a page is closed to us, the answer is that we do not have that data.
How to tell it to stop
Add this to your robots.txt and we will stop on our next visit. There is no override flag anywhere in our code, so this is not a request we can decide to ignore.
User-agent: LinkedAffiliateBot
Disallow: /How often it visits
At most one request every two seconds to any single host, and slower than that if your robots.txt sets a Crawl-delay. We read a handful of conventional paths, and a link from your homepage if it is labelled as an affiliate page. We do not crawl the rest of your site.
What it stores
Facts about the offer, and where we read them: commission rate, cookie window, payout terms, the network, plus the source URL and the time we fetched it. We do not copy your marketing copy, and we do not guess. A rate we cannot read without ambiguity is stored as unknown and shown as "Not published", never as an estimate.
If we got something wrong
Claim your listing. It is free, it takes proof that you run the domain, and after that the figures you correct are yours. Our crawler will not overwrite a field a company has corrected, on any later run.