[Haskell-cafe] PSA: If you're serializing floating point numbers, don't use binary

[Francesco Mazzoli]

`cereal` and `bytestring` `Builder` work by encoding to a big-endian `Word64`. A bit more expensive than a blind `unsafeCoerce`, but same idea.

Francesco

> On 9 Feb 2016, at 11:40, Tom Nielsen <tomn at beautifuldestinations.com> wrote:
> 
> I unsafeCoerce to Word64 first. This usually helps in making the output compatible with non-Haskell applications and is much faster.
> 
> Tom
> 
> On Tue, Feb 9, 2016 at 10:19 AM, Francesco Mazzoli <f at mazzo.li <mailto:f at mazzo.li>> wrote:
> Serialization of floating point numbers with binary is fantastically slow and incorrect if you’re using NaNs, see
> 
> https://github.com/kolmodin/binary/issues/64 <https://github.com/kolmodin/binary/issues/64>
> https://github.com/kolmodin/binary/issues/69 <https://github.com/kolmodin/binary/issues/69>
> I recently spent half a day debugging performance problems because of this, and since backwards compatibility with older formats is required, this problem is probably not going to be solved.
> 
> We decided to switch to cereal for this reason. With some patches <https://github.com/GaloisInc/cereal/pull/46> cereal was 30x faster for the data we were serializing (scientific computing, mostly Doubles packed in nested records containing vectors).
> 
> The size of the serialized data is also roughly 3 times smaller – with binary a Double takes at least 25 bytes of space instead of 8. With Float it’s even worse, 25 bytes instead of 8.
> 
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org <mailto:Haskell-Cafe at haskell.org>
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe <http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe>
> 
> 
> 
> 
> -- 
> Tom Nielsen | Chief Data Science Officer  
> Soho works | Tea building | 56 Shoreditch High St | London E1 6JJ
> +44 7961 270 416
> 
> Beautiful Destinations | The world’s largest travel influencer on Instagram
> Destinations <https://instagram.com/beautifuldestinations/> | Hotels <https://instagram.com/beautifulhotels> | Cuisines <https://instagram.com/beautifulcuisines> | Matters <https://instagram.com/beautifulmatters>
> www.beautifuldestinations.com <http://www.beautifuldestinations.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160209/bc8d8aea/attachment.html>