GHC Manuals online

[Claus Reinke]

> Google has never indexed all the mailing lists on haskell.org properly.  I never 
> found out why, although I believe John Peterson looked into it at one stage and 
> concluded that it wasn't easy to fix.  Google does index the mirrors though 
> (mailarchive.com, gmane.org).

try googling for "albus dumbledore tutorial haskell". i get 32 results, none 
from haskell.org. something is very wrong there.

checking the simplest things first, here is www.haskell.org/robots.txt:

    User-agent: *
    Disallow: /
    
    User-agent: *
    Disallow: /pipermail/

that puts an end to list archive search by any decent search engine's spiders
right-away, doesn't it? in fact, the first entry makes me wonder how we get
any results at all from haskell.org?

    http://en.wikipedia.org/wiki/Robots.txt

i don't know how the different wiki formats and redirections affect things,
or what the different servers at haskell.org have to say about this. and i know
that html pages at haskell.org use the robots meta tag for fine control, but 
i don't know how robots.txt and robots meta tag interact. i suspect that with
that particular robots.txt, google will index nothing, unless someone else
points directly at a page, and that page's meta tag allows indexing.

cleaning up that robots.txt should help, but someone must have put it
there for a reason?

claus