[Haskell-cafe] Updating haskell.org robots.txt

Justin Bailey jgbailey at gmail.com
Thu Jun 28 00:42:29 EDT 2007


All,

Referencing this thread (
http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html), I have
prepared a small update to robots.txt. It doesn't do much, but should
exclude all dynamic pages from being spidered. Please vet it and let me know
of any concerns. Note I'm not really interested in reconfiguring the wiki to
have nicer URLs - what is there is what I'm working with right now.

Here's the updated robots.txt (availabe as
http://www.haskell.org/robots2.txt right now):

  User-agent: *
  Disallow: /haskellwiki/Special:
  Disallow: /haskellwiki/Special%3A
  Disallow: /haskellwiki/?
  Disallow: /haskellwiki/%3f

Any feedback, let me know. Thanks!

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/haskell-cafe/attachments/20070627/eea2816f/attachment.htm


More information about the Haskell-Cafe mailing list