[Haskell-cafe] monoid fold concurrently
frederic-emmanuel.picca at synchrotron-soleil.fr
Fri Nov 15 21:29:58 UTC 2019
> > Is the list is guaranteed non-empty? Do you want to enforce that at the type
> > level (compile time), or "fail" at runtime when it is?
> > You should probably be a bit more explicit about what you mean by
> > "concurrently". Do you know in advance that the list length is sufficiently
> > short to make it reasonable to immediately fork an async thread for each?
the list can have 100000 images of 1 million of pixels, and it is not that short per image.
> Is that at least one? O(1) per core, ... hundreds, tens of thousands?
for now I have 100000 images to merge on 24 core.
> Do you have space constraints on the number of not yet folded
> together cubes that can be in memory at the same time?
On my computer I have 256 Go of memory enought to load all the images first, but I need to target smaller computer.
the biggest stack that I processed was 60Go for a final cube of 2000x2000x2000 of int32.
but most of the time it will be around 1000x1000x1000 of int32
> Is it actually a Monoid (has an identity), or only a Semigroup?
I can have a identiy which is an empty cube.
> Which is the (more) expensive operation, computing a cube or merging
> two already computed cubes?
computing the cube is constant in time for a given image size.
merging cube is slower and slower with time, since the cube grows from merge to merge.
> That is still ultimately order preserving (but associative): (a1 + a2) + (a3 + a4).
> Is the semigroup also commutative, would: (a2 + a4) + (a1 + a3) also work?
More information about the Haskell-Cafe