[Haskell-cafe] [SOLVED] Writing and testing a Storable instance for a record wrapping ForeignPtrs

[Viktor Dukhovni]

On Mon, Nov 28, 2022 at 08:42:40PM +0100, Hécate wrote:

> Thanks for the knowledge regarding the size of the ByteString!
> 
> Eventually the outside serialisation isn't up to me, but rather the 
> consumers. I have one potential user who told me they need the signature 
> and the message to be separate for the protocol they are implementing.
> 
> I guess if I defer to ByteString early on it'll be easier for the 
> library to be adapted to whatever use case comes after.

Note that "Lazy" Bytestrings are allocated as a chain of component
ByteString chunks, and their total length is an Int64 on all platforms.
The chunks might still be in memory, and in some cases could be truly
incrementally loaded from the underlying source.

However true streaming support is perhaps best via ByteStream from e.g.
https://hackage.haskell.org/package/streaming-bytestring-0.2.4/docs/Streaming-ByteString-Char8.html
which supports monadic incremental consumption (the underlying Monad
need not be IO, so pure sources are also supported).

The design space is wide.  If signatures are "detached" (available
independently of the message), and messages could be too large to
fit in memory, then streaming the message may be attractive.  But
then the protocol to the consumer needs to support streaming input
(e.g. HTTP PUT or POST with chunked transfer encoding).

With large messages one might even want chunk-level signatures that
authenticate all the chunks transmitted so far in order, with a final
empty chunk authenticating the entire message.  There should be a
standard format for this, but I am not aware of one.

CMS sadly does not suffice.  For example, messages that are just signed,
but not cryptographically tied to a particular transaction (and
recipient) are potentially subject to replays.  This may or may not
be OK.

A message with a clear transaction context (that makes out of context
replays detectable), encrypted to a given set of recipients, and then
signed has stronger security properties than just a signed blob that
is not tied to a clear context or recipient.

It is sadly common to use cryptography pixie dust to add security
theatre to application data flows without a careful analysis of
the security properties required and attained.

Security is difficult and brittle. :-(

-- 
    Viktor.