[Haskell-cafe] Updating haskell.org robots.txt
jgbailey at gmail.com
Thu Jun 28 00:42:29 EDT 2007
Referencing this thread (
http://www.haskell.org/pipermail/haskell-cafe/2007-May/025769.html), I have
prepared a small update to robots.txt. It doesn't do much, but should
exclude all dynamic pages from being spidered. Please vet it and let me know
of any concerns. Note I'm not really interested in reconfiguring the wiki to
have nicer URLs - what is there is what I'm working with right now.
Here's the updated robots.txt (availabe as
http://www.haskell.org/robots2.txt right now):
Any feedback, let me know. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Haskell-Cafe