[Haskell-cafe] [Haskell] A few days to go before the old server goes down
gale at sefer.org
Sun Jan 30 11:56:35 CET 2011
Henk-Jan van Tuyl wrote:
> I am trying to fetch the listed pages with commands like:
> wget -m http://oldhaskell.cs.yale.edu/hmake/
> but I only get the index.html files in the directories; what am I doing
The robots.txt file on the site is telling wget that downloading
the site in an automated way is not allowed. So wget stops.
Normally you should never tell wget to ignore the robots.txt,
as you could be damaging someone's web site. However,
in this case, since you are actually trying to rescue that very
web site, you can do so as follows:
wget -e robots=off -m ...
Hope this helps,
More information about the Haskell-Cafe