Facebook and Crawl-delay in Robots.txt?

Question

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?

Don't dupe questions across sites. You got your answer over there, by the way. (See Jeff's bio.)

Frank · Answer 1 · 2012-11-13 09:31:13Z

up vote -1 down vote

Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API. .

answered Nov 13 '12 at 9:31

Frank
700417

	Not a bot, a scraper. What evidence do you have for your claim? There's data disputing your claim here: stackoverflow.com/a/9165191/63094 – artlung Nov 13 '12 at 17:25
	Evidence, is 100s of log files. If you dont use facebook on your site, facebook will bot bot or spider it. The answer of Stack overflow is exactly the same. Facebook connects to your site when it is shared or if you use facebook FBML or other facebook code. – Frank Nov 20 '12 at 4:32

1 Answer