1. A program that reads web pages from the server over Internet, simulating browser behavior. 2. A (usually testing) tool that simulates human input events. 3. A programmable hardware machine.
36
votes
8answers
11k views
Static files in Flask - robot.txt, sitemap.xml (mod_wsgi)
Is there any clever solution to store static files in Flask's application root directory.
robots.txt and sitemap.xml are expected to be found in /, so my idea was to create routes for them:
...
8
votes
4answers
608 views
Googlebots Ignoring robots.txt? [closed]
I have a site with the following robots.txt in the root:
User-agent: *
Disabled: /
User-agent: Googlebot
Disabled: /
User-agent: Googlebot-Image
Disallow: /
And pages within this site are ...
7
votes
6answers
384 views
7
votes
2answers
291 views
Are there any CRobots style games that support robots written in more than one language?
Many years ago, just as I was getting started with programming, I ran into some programming games in the style of CRobots (I don't think it actually was CRobots, but a clone of sorts) which were ...
6
votes
3answers
3k views
Robots.txt: allow only major SE
Is there a way to configure the robots.txt so that the site accepts visits ONLY from Google, Yahoo! and MSN spiders?
6
votes
8answers
277 views
Running a piece of code for a certain amount of time in C#?
I am working on a robot that is capable of motion detection using a webcam. I am doing this in C#
The robot moves too fast, so I want to turn it on/off at short time intervals in order to reduce its ...
6
votes
3answers
2k views
Microsoft Robotics: cheap but very extensible robot?
Is there any cheap and very extensible robot kit, which can work with Microsoft Robotics?
I want to have a great choice of cool parts for a robot to buy. :)
If where is no such robot kit which can ...
5
votes
5answers
2k views
Should I get rid of bots visiting my site?
I have been noticing on my trackers that bots are visiting my site ALOT. Should I change or edit my robots.txt or change something? Not sure if thats good, because they are indexing or what?
5
votes
4answers
1k views
How do I stop search engines indexing a maintenance page
I need to setup a maintenance page for a website I'm running, e.g. for display when I'm performing site maintenance (scheduled downtime) or if something really breaks and I need to put up a holding ...
5
votes
6answers
12k views
What's a good Web Crawler tool [closed]
I need to index a whole lot of webpages, what good webcrawler utilities are there? I'm preferably after something that .NET can talk to, but that's not a showstopper.
What I really need is something ...
4
votes
4answers
1k views
Potential field method : Real Robots
Potential field method is a very popular simulation for Robot Navigation. However, has anyone implemented Potential field method on real robots ? Any reference or any claim of using the method in real ...
4
votes
6answers
1k views
PHP Detecting bot-like behavior
I am attempting to build a system that only shows users a CAPTCHA when bot-like behavior is detected. Here are the behaviors that I am currently looking for when somebody is filling out a contact ...
3
votes
6answers
1k views
How to Verify whether a Robot is Entering Information
I have a web form which the users fill and the info send to server and stored on a database. I am worried that Robots might just fill in the form and I will end up with a database full of useless ...
3
votes
4answers
353 views
How to show effectiveness of Erlang for programming Robots?
I am currently pursuing Masters in Embedded and for my thesis I have to study the effectiveness of Erlang for programming Robot. AFAIK Erlang's declarative nature and concurrency can be effective, so ...
3
votes
2answers
2k views
Allowing Google to bypass CAPTCHA verification - sensible or not?
My web site has a database lookup; filling out a CAPTCHA gives you 5 minutes of lookup time. There is also some custom code to detect any automated scripts. I do this as I don't want someone data ...
