[Haskell-cafe] Encrypting streamed data

[Viktor Dukhovni]

> On Jul 6, 2017, at 12:58 AM, Ivan Lazar Miljenovic <ivan.miljenovic at gmail.com> wrote:
> 
> I have a use case for needing to use public key cryptography to
> encrypt a large amount of data in a streaming fashion (get it out of a
> DB, encrypt, put into an AWS S3 bucket).

What are the data-format requirements?  Do you need (binary) CMS output?
GPG-compatible output?  Or just roll your own?

Integrity protection can be tricky with large data streams.  Most data
formats for enveloped data have a single MAC at the end, which means
that the decoder has to consume all the data before it is known to be
valid!

So if you're in a position to avoid a standard all-in-one format, it
makes sense to "packetize" the stream, with integrity protection for
each "packet", and packet sequence numbers to preserve overall stream
integrity.  With vast amounts of data, you'll want to be careful with
the symmetric cipher modes, AEAD (AES-GCM, for example) protects only
a limited amount of data before you need to rekey.  It may be simplest
to just generate a new symmetric key for every N megabytes of data.

With a careful design of the "packet" format, you can use in-memory
crypto for each packet.  Don't forget to include an "end-of-stream"
packet to defeat truncation attacks.

-- 
	Viktor.