Googlebot is the bot software that Google uses to crawl over 20 billion pages each day, the data obtained during a crawl is then analyzed and ranked on Google Search.
3
votes
3answers
112 views
Are penalties applied if you have too many 404 errors on your site?
Do the search engines take the number of 404 errors returned by a site into account when calculating the site rank? For example I have 8,000 404 errors and I am thinking that that's affecting my SEO ...
0
votes
2answers
40 views
How long does it take for Google to remove page from index? Or, why aren't these pages excluded?
A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
1
vote
2answers
53 views
How much time does Googlebot take to crawl the page after submitting it to index
I've a daily changing job site and I've also setup a dynamically generated XML sitemap for the search engines. But Googlebot indexes the at it's own priority.
I've a page linking to newly added jobs ...
1
vote
1answer
39 views
Screen resolution of Googlebot mobile?
Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it?
It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
43 views
Server overhead caused by bots?
I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html).
With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
39 views
Google not crawling my website [duplicate]
I own a website http://citrusbug.com.
We did not have proper content before so to prevent search engine from crawling our site
we added a robots.txt with code to prevent crawling.
Now we had ...
0
votes
1answer
23 views
Google attempting crawl of URLs with parameter randomly placed in URL
We have a weird SEO issue;
In Google Webmaster Tools we are seeing a huge spike in Not Found Server Errors.
There are a set of several hundred dynamic pages which we need to maintain (though not ...
0
votes
1answer
33 views
Google visits on Statcounter. How?
In my website’s Statcounter report, I found page views from Google. I thought it may be a Google bot’s visit but I found that particular visit comes through a search query (example.com+keyword) and it ...
1
vote
3answers
129 views
New or same URL on reactivation of user generated content?
We run a classified ad website.
Our users can temporarily deactivate their classified ads.
From a SEO point of view, when they reactivate it, should the ad use the same URL or use a new one?
Each ...
2
votes
1answer
74 views
How to have Google crawl pages that are only up for a week
We have flash sale product pages that are only up for 7 days and then go down. What's the best way to have google crawl these pages properly? The URL for these pages go up and down every week and are ...
2
votes
1answer
63 views
Can I use meta refresh instead of 301 redirect?
I have submitted my site with ugly URLs to Google, and now they are appearing as mysite.com/show.php?id=42.
Now, I know that I can 301-redirect it to mysite.com/article/42. However, my site uses the ...
2
votes
2answers
102 views
How can I block visitors located in the U.S. while still allowing the Googlebot?
Due to regulation and legal issues we cannot allow U.S. based visitors to access our website. For this reason we redirect all our U.S. traffic to a specialized page.
While we do have these ...
0
votes
1answer
37 views
How can I create a sitemap for 2000 products powered by a single php page?
Usually I generate sitemaps using www.xml-sitemaps.com and the limit is 500 pages. My website has barely 30, but I have a lot of products in my database, so the page product.php becomes ...
0
votes
0answers
32 views
Strange Google crawler 404s on Magento store involving “image” and “id” in URL
We're hit with over 10,000 404s on Google Webmaster all of a sudden. We've been getting them a few now and then, but not even close to this magnitude. Upon examining the items in this list of errors, ...
1
vote
2answers
64 views
Does specifying full webpage path in robots.txt would affect my website?
It may be a silly question but I need to clarify my doubt because it's related to robots.txt.
I need to prevent some label path of my website using robots.txt file.
User-agent: *
Disallow: ...
