Wonderful, powerful information, useful to anyone running a web site. Daring Fireball is being abused by these very same robots, most especially NameProtect (whose bots have hit this site over 1,100 times so far this month) and the unnamed weaselly spambot harvester whose user-agent string contains “Microsoft URL Control” (which at one point on February 21 hit this site 92 times in a single minute).
Since 1977, I have been opposed to the policy of banning robots. But I now have a new perspective on this:
INTERIOR: TATOOINE -- MOS EISLEY -- CANTINA. The young adventurer and his two mechanical servants follow Ben Kenobi into the smoke-filled cantina. The murky, moldy den is filled with a startling array of weird and exotic alien creatures and monsters at the long metallic bar. At first the sight is horrifying. One-eyed, thousand-eyed, slimy, furry, scaly, tentacled, and clawed creatures huddle over drinks. Ben moves to an empty spot at the bar near a group of repulsive but human scum. A huge, rough-looking Bartender stops Luke and the robots. BARTENDER We don't serve their kind here! Luke still recovering from the shock of seeing so many outlandish creatures, doesn't quite catch the bartender's drift. LUKE What? BARTENDER Your droids. They'll have to wait outside. We don't want them here. Luke looks at old Ben, who is busy talking to one of the Galactic pirates. He notices several of the gruesome creatures along the bar are giving him a very unfriendly glare. Luke pats Threepio on the shoulder. LUKE Listen, why don't you wait out by the speeder. We don't want any trouble. THREEPIO I heartily agree with you sir. Threepio and his stubby partner go outside and most of the creatures at the bar go back to their drinks.
(Source: Star Wars Episode IV: A New Hope, by George Lucas.)