Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 1 down vote favorite

In my logs, I keep noticing that 69.58.178.57, which is Verisign, is spidering through every page on my website, excluding pages of forms. Does anybody know why they are doing this?

share|improve this question
1  
Why is this question tagged with SSL/SSL-certificate? – Lèse majesté Jan 6 at 3:04
Probably need a bit of log detail on this one to see what's really going on. – Fiasco Labs Jan 6 at 4:01
If you're not using their services(see response from @mahnsc), it's probably just going to come down to "because they want to" and you'll have to ask if you really want to know. Anyone can run a spider. Years ago, a personal site I ran was regularly being hit by the IAEA for reasons I couldn't even begin to speculate about. – Su' Jan 10 at 14:36

2 Answers

up vote 1 down vote

It could be related to the daily malware scan[CNet, PDF @Verisign] they run automatically as a service to their ssl certificate customers. You could always opt out of the service temporarily in order to see if the crawls from that IP cease.

share|improve this answer
up vote 0 down vote

Other people have experienced similar things: http://www.forumpostersunion.com/showthread.php?t=4163

Specifically highlighting the last post, where the bot appears to be spidering things high in SEO ranking. My personal opinion, is that someone is spidering websites and in fact spoofing their IP to hide who they are. I don't believe VeriSign rents out infrastructure or servers, so either they have been compromised to someone is putting the blame on them.

My personal recommendation, is to simply ban the bot (not using a robots.txt file since I'd almost guarantee it ignores it) by IP address (some people report it using the entire range 69.58.178.* and not to worry too much about it unless it's slowing your website down.

share|improve this answer
1  
Unless the attacker is on the same subnet as you and sniffing all IP traffic, IP spoofing only works one way. You can send spoofed packets, but then the responses won't come back to you. As such, it's pretty difficult to even open a TCP connection while spoofing your IP on the internet, much less try to spider a website (which requires actually reading the responses from the web server). – Lèse majesté Jan 6 at 3:03
I understand it works only one direction. It is possible to send GET requests via an invalid IP. But we don't know where it finds the pages to spider. If the pages trend towards popularity before being crawled, perhaps the bot is cross-referencing with search engines. – ionFish Jan 6 at 3:07
1  
To send a GET request, you'd need to open a TCP connection, which requires a SYN/SYN-ACK/ACK handshake with correct sequence numbers, which is pretty close to impossible for the attacker to perform without seeing the random sequence number the sever chose for the SYN-ACK. And if you can't actually capture the server response to the GET request, how are you really "spidering" the site? – Lèse majesté Jan 6 at 3:14
IP Spoofing is highly overrated and best used for DDOS backscatter and joe jobbing where you want the replies to be directed to the spoofed IP. It doesn't work the way you think it works. – Fiasco Labs Jan 6 at 3:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.