[IP] more on AFP Sues Google News
------- Original message -------
From: Carl Malamud <carl@xxxxxxxxx>
Sent: 20/3/'05, 12:02
>
> ------ Forwarded Message
> From: Dana Blankenhorn <dana@xxxxxxxxxx>
> Date: Sun, 20 Mar 2005 14:08:04 -0500
> To: <dave@xxxxxxxxxx>
> Subject: Re: [IP] AFP Sues Google News
>
<snip>
> Now to Google's defense.
>
> Exhibit A for the defense. This is an Agence France-Presse story published
> on its customer site, Velo News. It has been spidered by Google News,
> obviously without the express written permission of Agence France-Presse.
>
> But is it possible for Google News not to spider this story? Yes, it is.
> That would require only AFP to include a robots.txt file on stories it sends
> affiliates, instructing those pages not to allow spiders or robots to see
> them.
Their site shows a robots.txt file in place for at least a month:
wget --save-headers http://www.afp.com/robots.txt
HTTP/1.1 200 OK
Date: Sun, 20 Mar 2005 19:58:28 GMT
Server: Apache/1.3.27 (Unix)
Cache-Control: max-age=300
Expires: Sun, 20 Mar 2005 20:03:28 GMT
Last-Modified: Wed, 23 Feb 2005 10:54:38 GMT
ETag: "761b2-4f-421c60ee"
Accept-Ranges: bytes
Content-Length: 79
Connection: close
Content-Type: text/plain
User-Agent: *
Disallow: /beta
Disallow: /francais/news
Disallow: /english/news
And, Archive.org shows that they've had that in place for a long
time before:
http://web.archive.org/web/*/http://www.afp.com/robots.txt
Regards,
Carl
-------------------------------------
You are subscribed as roessler@xxxxxxxxxxxxxxxxxx
To manage your subscription, go to
http://v2.listbox.com/member/?listname=ip
Archives at: http://www.interesting-people.org/archives/interesting-people/