[lbo-talk] WhiteHouse.gov blocks search engines

Doug Henwood dhenwood at panix.com
Tue Oct 28 11:59:07 PST 2003


Date: Tue, 28 Oct 2003 11:28:03 -0700 Subject: Whitehouse.gov blocks search engines From: Matt Herr <mherr at wyoming.com>

WHITE HOUSE WEB SITE BLOCKS SEARCH ENGINES

The White House is blocking search engines from accessing much of the White House web site, particularly web pages relating to Iraq. As a result, these web pages cannot be directly accessed through external search engines such as google.com.

The White House employs a "robots.txt" file which "disallows" search engine access to over a thousand specified directories. See:

<http://www.whitehouse.gov/robots.txt>

The matter was first noticed and discussed on this web page:

<http://www.bway.net/~keith/whrobots/>

Say, for example, that you want to find the White House web page where President Bush made this remarkable statement on October 3:

"See, free nations are peaceful nations. Free nations don't attack each other. Free nations don't develop weapons of mass destruction."

You can't get there from Google, because the page is blocked. But it's here:

<http://www.whitehouse.gov/news/releases/2003/10/20031003-4.html>



More information about the lbo-talk mailing list