Avete notato che ora hanno “Bloccato” ogni richiesta di robots.txt ?
Mi da un 404… su
Richiesta:
(Request-Line) GET /robots.txt HTTP/1.1
Host http://www.innovazionepa.gov.it
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0) Gecko/20100101 Firefox/4.0
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language it-it,it;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding gzip, deflate
Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive 115
Proxy-Connection keep-alive
DNT 1
—————–
Risposta:
(Status-Line) HTTP/1.1 404 Not Found
Content-Type text/html
Server Microsoft-IIS/6.0
X-Powered-By ASP.NET
Date Thu, 31 Mar 2011 14:25:19 GMT
Content-Length 1635
Proxy-Connection Keep-Alive
Connection Keep-Alive
Age 0
——–