Re: Why prevent the mailing list from being indexed?

["Daniel Shahaf" <d.s@xxxxxxxxxxxxxxxxxx>]

Stephane Chazelas wrote on Mon, 20 May 2019 10:30 +00:00:
> I was surprised when a google search on the zsh mailing list
> archives turned up nothing when I knew it should (site:zsh.org
> ZSH_XTRACEFD) and then saw:
> 
> https://www.zsh.org/robots.txt
> 
> User-agent: *
> Disallow: /cgi-bin/
> Disallow: /mla/zsh-announce/
> Disallow: /mla/zsh-users/
> Disallow: /mla/zsh-workers/
> Disallow: /pub/old/
> 
> Why?
> 
> Having google/duckduckgo searches return results from there
> would be very useful,

Note that it only disallows the directories that contain the raw rfc822
files.  The rendered HTML indexes at /mla/workers/ aren't disallowed,
so they should be implicitly allowed, shouldn't they?

Cheers,

Daniel

> especially considering that the mailing
> list is also used as an issue tracker and to some extent as a
> changelog.
> 
> I know (well, I just discovered) it can be queried with
> https://www.zsh.org/cgi-bin/mla/wilma_glimpse/workers?query=ZSH_XTRACEFD
> but you have to know it's there and look explicitely in the ml
> as opposed to generally on the web.