Who needs and wants parallelism-friendly core libs?

Fri Nov 22 00:00:54 UTC 2013

Since vector is needed to build ghc with dph, and has gone fallow, I
suppose it really does need a maintainer.

I've personally spent a lot of time mucking around in its internals
building hybrid-vector and vector-instances and many of my recent projects
center on the guts of vector.

To that end, I've forked vector over to my github account.

If nobody has any objections, we can use that as a sort of staging ground
for this sort of change before they filter up to the main repository.

I'm happy to juggle whatever integration work is required as an individual
maintainer, or to work through the committee if folks really prefer, just
to make sure everything continues to work well, but I think things like the
recent change to SPEC in HEAD, future SIMD integration via generalized
stream fusion using Geoffrey's work, and this CAS proposal will require
somewhat active management.

That said, if someone else feels strongly about wanting to take over
maintainership personally (Geoff?), I'm happy to have the conversation.

-Edward

On Thu, Nov 21, 2013 at 6:37 PM, Edward Kmett <ekmett at gmail.com> wrote:

> I have several bit-vector variants floating around in various
> side-projects, such as github.com/ekmett/succinct and
> github.com/ekmett/structures but I don't currently have HBVs per se. I
> have some code that rolls up aggregations by that same complete tree
> structure though to build range-min-max trees and the like.
>
>
> On Thu, Nov 21, 2013 at 4:00 PM, Ryan Newton <rrnewton at gmail.com> wrote:
>
>> Yep, it is public!  So it is fine to do this to rip the MutableByteArray
>> out of the relevant "MVector s a" types:
>>
>>
>> https://github.com/rrnewton/haskell-lockfree/blob/cc2690f96ed616f9d1221770c5213d1ae41facc9/vector-atomics/Data/Vector/Unboxed/Atomic.hs
>>
>> And then we can build up a "vector-atomics" package that exposes all the
>> relevant ops on [un]boxed vectors (and Storable too if we want to use the
>> bits-atomic package).
>>
>> Personally, right this moment I need fetch-and-OR and fetch-and-AND
>> because I want a concurrent hierarchical bit-vector for storing subsets of
>> the integers [0,N).  Edward, I don't suppose you have that one up your
>> sleeve already ;-)?
>>
>>   -Ryan
>>
>>
>>
>> On Thu, Nov 21, 2013 at 11:29 AM, Edward Kmett <ekmett at gmail.com> wrote:
>>
>>> Actually, the Primitive.Vector constructor should be irrelevant.
>>> Nevermind. It is the mutable one you need and that one is public.
>>>
>>>
>>> On Thu, Nov 21, 2013 at 11:11 AM, Edward Kmett <ekmett at gmail.com> wrote:
>>>
>>>> I'd like to hash this out a bit further. As this issue affects vector primarily,
>>>> it is outside of the scope of the core library committee per se, and is
>>>> largely up to Roman as the maintainer of vector.
>>>>
>>>> That said if I put on my personal and not my 'committee' hat:
>>>>
>>>> I would use these operations, personally, were they available to me.
>>>>
>>>> The question is how to best expose them from an API perspective.
>>>>
>>>> Vector currently exposes enough (or at least almost enough) of the
>>>> implementation details that such support could be implemented 'out-of-line'
>>>> from the main library. (I think the Data.Vector.Primitive constructor is
>>>> currently not exported, and you'd need that.)
>>>>
>>>> I don't *think* Roman would object too strenuously to exporting it
>>>> though and I've personally wanted it for other reasons. This would open the
>>>> door to this sort of thing being something you might supply entirely
>>>> outside of the library, giving the final placement of the code some
>>>> flexibility.
>>>>
>>>> You could handle CAS operations for Primitive and boxed Vectors pretty
>>>> easily at least, leaving off Unboxed and my various hybrid vector
>>>> approaches as they are intrinsically tied to working on multiple underlying
>>>> source vectors.
>>>>
>>>> I'm pretty open to ideas about where the instances should live though.
>>>> One argument for putting it in vector is you could extend Prim with the CAS
>>>> operations directly rather than subclass it. An argument against is it
>>>> could then iterate on a more flexible time table.
>>>>
>>>> -Edward
>>>>
>>>>
>>>> On Thu, Nov 21, 2013 at 10:30 AM, Ryan Newton <rrnewton at gmail.com>wrote:
>>>>
>>>>> [Popping up to the big-picture question for a moment, from the
>>>>> previous thread on Vector CAS.]
>>>>>
>>>>>  Given that a lot of people are really advertising Haskell on the
>>>>> parallel/concurrent front, it seems like we *should* be committed as
>>>>> a community to making our basic libraries support parallel/concurrent use
>>>>> cases effectively.  This will necessarily mean touching some core libraries
>>>>> (container, vector...).
>>>>>
>>>>> The two things that have come up for me recently are (1) CAS/RMW on
>>>>> [unboxed] Vectors and a general notion of balanced-splitting for tree-based
>>>>> structures (Map,Set).  But I'm sure there are others!
>>>>>
>>>>> Is any one else invested in getting this kind of thing moving?  It
>>>>> would be great to have a partner.  I can commit a small amount of cycles,
>>>>> but I don't have lots of bandwidth for non-research infrastructure work
>>>>> presently.  Edward, is any of this useful to you?
>>>>>
>>>>> In many of these cases it seems like the main problem is not a
>>>>> technical one, but the social one of consensus building (and fighting
>>>>> excessive "stop energy" if it appears).  Edward, as chair of the core
>>>>> libraries committee, maybe you could help with this?
>>>>>
>>>>> Cheers,
>>>>>   -Ryan
>>>>>
>>>>>
>>>>> On Thu, Nov 21, 2013 at 10:25 AM, Ryan Newton <rrnewton at gmail.com>wrote:
>>>>>
>>>>>> [@Aleksey] Very good point!  I'd missed that vector does the SOA->AOS
>>>>>> business too.  (I thought that was just Repa & Accelerate.)
>>>>>>
>>>>>> In any case, an unboxed vector of tuples can't *actually* support
>>>>>> atomic CAS anyway, so not supporting it is inevitable.
>>>>>>    It does mean that exposeMutableByteArray can't be a method of the
>>>>>> Unbox class.  It would need to be another class capturing the subset of
>>>>>> vector types that are "Atomic" (support concurrent/atomic memory ops).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/libraries/attachments/20131121/695a3d69/attachment.html>