Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 0 down vote favorite

is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!

share|improve this question
I found this site to be pretty thorough when I was initially researching robots.txt robotstxt.org – davidisawesome Jun 5 '12 at 19:44
didn't have what i was looking for, just need to know if i should disallow the age gate page?? – Jordan Reiser Jun 5 '12 at 19:58
personally I don't see any reason you should have to. The age gate just has the user confirm they are at least x-years-old, then they continue to the home page, right? It sounds like a personal preference really – davidisawesome Jun 5 '12 at 20:27
right, but i read that sometimes the bots have trouble getting through to the home page because of the age gate – Jordan Reiser Jun 5 '12 at 20:27
hm.. I hadn't heard that before. Not saying it isn't true for maybe some crawlers, I just am not sure and there are a ton of different ones so I don't know if it would be a true or false answer across the board. If that is the case, just block the age gate page – davidisawesome Jun 5 '12 at 20:31
show 1 more comment

migrated from stackoverflow.com Jun 29 '12 at 12:48

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.