Tuesday, February 10, 2009

Story Post: At least until we get robots.txt fixed…

All bots, scripts and other user agents are free to scrape BlogNomic in an automated fashion.

(This information appears on http://blognomic.com/robots.txt. Admittedly, it isn’t in the right format, but we have to give the bots out there some clue as to if they’re allowed to or not…)

Comments

Wakukee:

10-02-2009 20:53:09 UTC

???
No bots!

arthexis: he/him

10-02-2009 20:54:20 UTC

Finally! I can play legally!

ais523:

10-02-2009 20:58:36 UTC

@Wakukee: This is basically just allowing scripts to read messages. robots.txt is meant to tell bots whether they’re allowed to scrape a site or not; for instance, you can use robots.txt not to be listed in Google results. robots.txt seems to be a copy of the Main Page at the moment, so I’m giving Google, and anyone else who wants to scrape BlogNomic, a bit of guidance.

Note that reading BlogNomic != writing to it; no bot would be able to write to it without getting a username, which would require solving the CAPTCHA on the login form and therefore human help. It would probably be violating the rules, too, for a bot to do something like that.

Kevan: he/him

10-02-2009 21:49:54 UTC

robots.txt is now fixed.