[Haskell-cafe] Ideas on a fast and tidy CSV library

[Justin Paston-Cooper]

Dear All,

I now have some example code. I have put it on: http://pastebin.com/D9MPmyVd.

vectorBinner is simply of type Vector Int -> Int. I am inputting a 1.5GB
CSV on stdin, and would like vectorBinner to run over every single record,
outputting results as computed, thus running in constant memory. My
programme instead quickly approaches full memory use. Is there any way to
work around this?

Justin


On 25 July 2013 17:53, Johan Tibell <johan.tibell at gmail.com> wrote:

> You can use the Incremental or Streaming modules to get more fine
> grained control over when new parsed records are produced.
>
> On Thu, Jul 25, 2013 at 11:02 AM, Justin Paston-Cooper
> <paston.cooper at gmail.com> wrote:
> > I hadn't yet tried profiling the programme. I actually deleted it a few
> days
> > ago. I'm going to try to get something new running, and I will report
> back.
> > On a slightly less related track: Is there any way to use cassava so
> that I
> > can have pure state and also yield CSV lines while my computation is
> running
> > instead of everything at the end as would be with the State monad?
> >
> >
> > On 23 July 2013 22:13, Johan Tibell <johan.tibell at gmail.com> wrote:
> >>
> >> On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss at gmail.com>
> >> wrote:
> >> > Justin Paston-Cooper <paston.cooper at gmail.com> writes:
> >> >
> >> >> Dear All,
> >> >>
> >> >> Recently I have been doing a lot of CSV processing. I initially tried
> >> >> to
> >> >> use the Data.Csv (cassava) library provided on Hackage, but I found
> >> >> this to
> >> >> still be too slow for my needs. In the meantime I have reverted to
> >> >> hacking
> >> >> something together in C, but I have been left wondering whether a
> tidy
> >> >> solution might be possible to implement in Haskell.
> >> >>
> >> > Have you tried profiling your cassava implementation? In my experience
> >> > I've found it's quite quick. If you have an example of a slow path I'm
> >> > sure Johan (cc'd) would like to know about it.
> >>
> >> I'm always interested in examples of code that is not running fast
> >> enough. Send me a reproducible example (preferably as a bug on the
> >> GitHub bug tracker) and I'll take a look.
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130821/3633b16a/attachment.htm>