[Haskell-cafe] Monad transformer performance - Request to review benchmarking code + results

Oliver Charles ollie at ocharles.org.uk
Sun Jan 29 14:20:35 UTC 2017


I would wager a guess that this can be solved with INLINE pragmas. We
recently added INLINE to just about everything in transformers and got a
significant speed up.

On Sun, 29 Jan 2017, 11:18 am David Turner, <dct25-561bs at mythic-beasts.com>
wrote:

> I would guess that the issue lies within HtmlT, which looks vaguely
> similar to a WriterT transformer but without much in the way of
> optimisation (e.g. INLINE pragmas). But that's just a guess after about 30
> sec of glancing at
> https://hackage.haskell.org/package/lucid-2.9.7/docs/src/Lucid-Base.html
> so don't take it as gospel.
>
> My machine is apparently an i7-4770 of a similar vintage to yours, running
> Ubuntu in a VirtualBox VM hosted on Windows. 4GB of RAM in the VM, 16 in
> the host FWIW.
>
>
> On 29 Jan 2017 10:26, "Saurabh Nanda" <saurabhnanda at gmail.com> wrote:
>
> Thank you for the PR. Does your research suggest something is wrong with
> HtmlT when combined with any MonadIO, not necessarily ActionT? Is this an
> mtl issue or a lucid issue in that case?
>
> Curiously, what's your machine config? I'm on a late 2011 macbook pro with
> 10G ram and some old i5.
>
> -- Saurabh.
>
> On 29 Jan 2017 3:05 pm, "David Turner" <dct25-561bs at mythic-beasts.com>
> wrote:
>
> The methodology does look reasonable, although I think you should wait for
> all the scotty threads to start before starting the benchmarks, as I see
> this interleaved output:
>
> Setting phasers to stun... (port 3002) (ctrl-c to quit)
> Setting phasers to stun... (port 3003) (ctrl-c to quit)
> Setting phasers to stun... (port 3001) (ctrl-c to quit)
> benchmarking bareScotty
> Setting phasers to stun... (port 3000) (ctrl-c to quit)
>
> Your numbers are wayyy slower than the ones I see on my dev machine:
>
> benchmarking bareScotty
> Setting phasers to stun... (port 3000) (ctrl-c to quit)
> time                 10.94 ms   (10.36 ms .. 11.52 ms)
>                      0.979 R²   (0.961 R² .. 0.989 R²)
> mean                 12.53 ms   (11.98 ms .. 13.28 ms)
> std dev              1.702 ms   (1.187 ms .. 2.589 ms)
> variance introduced by outliers: 66% (severely inflated)
>
> benchmarking bareScottyBareLucid
> time                 12.95 ms   (12.28 ms .. 13.95 ms)
>                      0.972 R²   (0.951 R² .. 0.989 R²)
> mean                 12.20 ms   (11.75 ms .. 12.69 ms)
> std dev              1.236 ms   (991.3 μs .. 1.601 ms)
> variance introduced by outliers: 50% (severely inflated)
>
> benchmarking transScottyBareLucid
> time                 12.05 ms   (11.70 ms .. 12.39 ms)
>                      0.992 R²   (0.982 R² .. 0.996 R²)
> mean                 12.43 ms   (12.06 ms .. 13.01 ms)
> std dev              1.320 ms   (880.5 μs .. 2.071 ms)
> variance introduced by outliers: 54% (severely inflated)
>
> benchmarking transScottyTransLucid
> time                 39.73 ms   (32.16 ms .. 49.45 ms)
>                      0.668 R²   (0.303 R² .. 0.969 R²)
> mean                 42.59 ms   (36.69 ms .. 54.38 ms)
> std dev              16.52 ms   (8.456 ms .. 25.96 ms)
> variance introduced by outliers: 92% (severely inflated)
>
> benchmarking bareScotty
> time                 11.46 ms   (10.89 ms .. 12.07 ms)
>                      0.986 R²   (0.975 R² .. 0.994 R²)
> mean                 11.73 ms   (11.45 ms .. 12.07 ms)
> std dev              800.6 μs   (636.8 μs .. 975.3 μs)
> variance introduced by outliers: 34% (moderately inflated)
>
> but nonetheless I do also see the one using renderTextT to be
> substantially slower than the one without.
>
> I've sent you a PR [1] that isolates Lucid from Scotty and shows that
> renderTextT is twice as slow over IO than it is over Identity, and it's
> ~10% slower over Reader too:
>
> benchmarking renderText
> time                 5.529 ms   (5.328 ms .. 5.709 ms)
>                      0.990 R²   (0.983 R² .. 0.995 R²)
> mean                 5.645 ms   (5.472 ms .. 5.888 ms)
> std dev              593.0 μs   (352.5 μs .. 908.2 μs)
> variance introduced by outliers: 63% (severely inflated)
>
> benchmarking renderTextT Id
> time                 5.439 ms   (5.243 ms .. 5.640 ms)
>                      0.991 R²   (0.985 R² .. 0.996 R²)
> mean                 5.498 ms   (5.367 ms .. 5.631 ms)
> std dev              408.8 μs   (323.8 μs .. 552.9 μs)
> variance introduced by outliers: 45% (moderately inflated)
>
> benchmarking renderTextT Rd
> time                 6.173 ms   (5.983 ms .. 6.396 ms)
>                      0.990 R²   (0.983 R² .. 0.995 R²)
> mean                 6.284 ms   (6.127 ms .. 6.527 ms)
> std dev              581.6 μs   (422.9 μs .. 773.0 μs)
> variance introduced by outliers: 55% (severely inflated)
>
> benchmarking renderTextT IO
> time                 12.35 ms   (11.84 ms .. 12.84 ms)
>                      0.989 R²   (0.982 R² .. 0.995 R²)
> mean                 12.22 ms   (11.85 ms .. 12.76 ms)
> std dev              1.159 ms   (729.5 μs .. 1.683 ms)
> variance introduced by outliers: 50% (severely inflated)
>
> I tried replacing
>
>     forM [1..10000] (\_ -> div_ "hello world!")
>
> with
>
>     replicateM_ 10000 (div_ "hello world!")
>
> which discards the list of 10,000 () values that the forM thing generates,
> but this made very little difference.
>
> Hope this helps,
>
> David
>
>
> [1] https://github.com/vacationlabs/monad-transformer-benchmark/pull/2
>
>
>
> On 29 January 2017 at 07:26, Saurabh Nanda <saurabhnanda at gmail.com> wrote:
>
> Hi,
>
> I was noticing severe drop in performance when Lucid's HtmlT was being
> combined with Scotty's ActionT. I've tried putting together a minimal repro
> at https://github.com/vacationlabs/monad-transformer-benchmark Request
> someone with better knowledge of benchmarking to check if the benchmarking
> methodology is correct.
>
> Is my reading of 200ms performance penalty correct?
>
> -- Saurabh.
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
>
>
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20170129/780d4217/attachment.html>


More information about the Haskell-Cafe mailing list