[Haskell-cafe] Simple HTTP lib for Windows?

[Daniel McAllansmith]

On Sunday 28 January 2007 09:14, Neil Mitchell wrote:
> Hi Alistair,
>
> > > Is there a simple way to get the contents of a webpage using Haskell on
> > > a Windows box?
> >
> > This isn't exactly what you want, but it gets you partway there. Not
> > sure if LineBuffering or NoBuffering is the best option. Line
> > buffering should be fine for just text output, but if you request a
> > binary object (like an image) then you have to read exactly the number
> > of bytes specified, and no more.
>
> This works great for haskell.org, unfortunately it doesn't work as
> well with the rest of the web universe.
>
> With www.google.com I get: Program error: <handle>: IO.hGetChar:
> illegal operation
>
> With www.slashdot.org I get: 501 Not Implemented returned
>
> www.msnbc.msn.com works fine.
>
> Any ideas why? 

At the very least it's missing the HTTP version on the request line, and you 
almost always need to send a Host header.

For a start you could try changing client to:

client server port page = do
  h <- connectTo server (PortNumber port)
  hSetBuffering h NoBuffering
  putStrLn "send request"
  hPutStrLn h ("GET " ++ page ++ " HTTP/1.1\r")
  hPutStrLn h ("Host: " ++ server ++ "\r")
  hPutStrLn h "\r"
  hPutStrLn h "\r"
  putStrLn "wait for response"
  readResponse h
  putStrLn ""

Note that I haven't tried this, or the rest of Alistair code at all, so the 
usual 30 day money back guarantee doesn't apply.  It certainly won't handle 
redirects.


> Are there any alternatives to read in a file off the 
> internet (i.e. wget but as a library)

The http library sort of works most of the time, but there are several bugs 
that cause it to fail on many 'in the wild' webservers.

HXT has a wrapper around a command line invocation of cURL.  It works better.  
There is still a problem with redirects, but thats an easy enough fix.
I doubt that it would be very easy to extract it from the surrounding HXT 
framework though.
It would be nice to have a binding to libcurl.

Daniel