Dicks n' Dough

Marco Anglesio mpa at the-wire.com
Fri Mar 15 07:38:18 PST 2002


On Fri, 15 Mar 2002, Kelley wrote:
> the list gets approx 100 posts a day. with all the cross referenced pages
> that come up in results, 8980 isn't very many result for a list that is
> archived back to May 1998. i know how to do a site search, since i

The simple answer is that I hadn't rebuilt the missing archive months the last time that google crawled the page. Google will deprecate those links as a result. It isn't the same as the list not being on google.

"spiderciding", as you colourfully putting, involves creating a robots.txt page which prevents the robot from crawling the server. If a not insignificant number of people would like me to create a robots.txt page and keep the archive from being indexed by google, then they can let me know. Surprisingly enough, I've heard not a peep in 2 years.

m.

,--------------------------------------------------------------------------.
> Marco Anglesio | People demand freedom of speech <
> mpa at the-wire.com | to make up for the freedom of <
> http://www.the-wire.com/~mpa | thought which they avoid. <
> | --Soren Kierkegaard <
`--------------------------------------------------------------------------'



More information about the lbo-talk mailing list