[Haskell-cafe] Monad transformer performance - Request to review benchmarking code + results

[David Turner]

The methodology does look reasonable, although I think you should wait for
all the scotty threads to start before starting the benchmarks, as I see
this interleaved output:

Setting phasers to stun... (port 3002) (ctrl-c to quit)
Setting phasers to stun... (port 3003) (ctrl-c to quit)
Setting phasers to stun... (port 3001) (ctrl-c to quit)
benchmarking bareScotty
Setting phasers to stun... (port 3000) (ctrl-c to quit)

Your numbers are wayyy slower than the ones I see on my dev machine:

benchmarking bareScotty
Setting phasers to stun... (port 3000) (ctrl-c to quit)
time                 10.94 ms   (10.36 ms .. 11.52 ms)
                     0.979 R²   (0.961 R² .. 0.989 R²)
mean                 12.53 ms   (11.98 ms .. 13.28 ms)
std dev              1.702 ms   (1.187 ms .. 2.589 ms)
variance introduced by outliers: 66% (severely inflated)

benchmarking bareScottyBareLucid
time                 12.95 ms   (12.28 ms .. 13.95 ms)
                     0.972 R²   (0.951 R² .. 0.989 R²)
mean                 12.20 ms   (11.75 ms .. 12.69 ms)
std dev              1.236 ms   (991.3 μs .. 1.601 ms)
variance introduced by outliers: 50% (severely inflated)

benchmarking transScottyBareLucid
time                 12.05 ms   (11.70 ms .. 12.39 ms)
                     0.992 R²   (0.982 R² .. 0.996 R²)
mean                 12.43 ms   (12.06 ms .. 13.01 ms)
std dev              1.320 ms   (880.5 μs .. 2.071 ms)
variance introduced by outliers: 54% (severely inflated)

benchmarking transScottyTransLucid
time                 39.73 ms   (32.16 ms .. 49.45 ms)
                     0.668 R²   (0.303 R² .. 0.969 R²)
mean                 42.59 ms   (36.69 ms .. 54.38 ms)
std dev              16.52 ms   (8.456 ms .. 25.96 ms)
variance introduced by outliers: 92% (severely inflated)

benchmarking bareScotty
time                 11.46 ms   (10.89 ms .. 12.07 ms)
                     0.986 R²   (0.975 R² .. 0.994 R²)
mean                 11.73 ms   (11.45 ms .. 12.07 ms)
std dev              800.6 μs   (636.8 μs .. 975.3 μs)
variance introduced by outliers: 34% (moderately inflated)

but nonetheless I do also see the one using renderTextT to be substantially
slower than the one without.

I've sent you a PR [1] that isolates Lucid from Scotty and shows that
renderTextT is twice as slow over IO than it is over Identity, and it's
~10% slower over Reader too:

benchmarking renderText
time                 5.529 ms   (5.328 ms .. 5.709 ms)
                     0.990 R²   (0.983 R² .. 0.995 R²)
mean                 5.645 ms   (5.472 ms .. 5.888 ms)
std dev              593.0 μs   (352.5 μs .. 908.2 μs)
variance introduced by outliers: 63% (severely inflated)

benchmarking renderTextT Id
time                 5.439 ms   (5.243 ms .. 5.640 ms)
                     0.991 R²   (0.985 R² .. 0.996 R²)
mean                 5.498 ms   (5.367 ms .. 5.631 ms)
std dev              408.8 μs   (323.8 μs .. 552.9 μs)
variance introduced by outliers: 45% (moderately inflated)

benchmarking renderTextT Rd
time                 6.173 ms   (5.983 ms .. 6.396 ms)
                     0.990 R²   (0.983 R² .. 0.995 R²)
mean                 6.284 ms   (6.127 ms .. 6.527 ms)
std dev              581.6 μs   (422.9 μs .. 773.0 μs)
variance introduced by outliers: 55% (severely inflated)

benchmarking renderTextT IO
time                 12.35 ms   (11.84 ms .. 12.84 ms)
                     0.989 R²   (0.982 R² .. 0.995 R²)
mean                 12.22 ms   (11.85 ms .. 12.76 ms)
std dev              1.159 ms   (729.5 μs .. 1.683 ms)
variance introduced by outliers: 50% (severely inflated)

I tried replacing

    forM [1..10000] (\_ -> div_ "hello world!")

with

    replicateM_ 10000 (div_ "hello world!")

which discards the list of 10,000 () values that the forM thing generates,
but this made very little difference.

Hope this helps,

David


[1] https://github.com/vacationlabs/monad-transformer-benchmark/pull/2



On 29 January 2017 at 07:26, Saurabh Nanda <saurabhnanda at gmail.com> wrote:

> Hi,
>
> I was noticing severe drop in performance when Lucid's HtmlT was being
> combined with Scotty's ActionT. I've tried putting together a minimal repro
> at https://github.com/vacationlabs/monad-transformer-benchmark Request
> someone with better knowledge of benchmarking to check if the benchmarking
> methodology is correct.
>
> Is my reading of 200ms performance penalty correct?
>
> -- Saurabh.
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20170129/d0bb7a93/attachment.html>