Ho fatto un test “camuffandomi da googlebot” per vedere se almeno lui lo accettano:
Richiesta:
—–
(Request-Line) GET /robots.txt HTTP/1.1
Host http://www.innovazionepa.gov.it
User-Agent Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language it-it,it;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Proxy-Connection keep-alive
DNT 1
—–
Risposta:
————–
(Status-Line) HTTP/1.1 404 Not Found
Content-Type text/html
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET
Date Thu, 31 Mar 2011 14:31:54 GMT
Content-Length 1635
Proxy-Connection Keep-Alive
Connection Keep-Alive
Age 0
—————-
niente… se non c’e’ robots.txt si naviga ovunque quindi?
