FFI: c/c++ struct on stack as an argument or return value

Tue Mar 18 12:19:00 UTC 2014

On 18/03/2014 01:37, Manuel M T Chakravarty wrote:
> Yuras Shumovich <shumovichy at gmail.com>:
>> I think the compiler is the right place. It is impossible to have
>> efficient implementation in a library.
>>
>> For dynamic wrapper (foreign import "wrapper" stuff) ghc generates piece
>> of executable code at runtime. There are native implementations for a
>> number of platforms, and libffi is used as a fall back for other
>> platforms (see rts/Adjustor.c). AFAIK it is done that way because libffi
>> is slower then native implementation.
>>
>> Library implementation can't generate native dynamic wrapper, it has to
>> use slow libffi.
>
> When we first implemented the FFI, there was no libffi. Maintaining the adjustor code for all platforms is a PITA; hence, using libffi was a welcome way to improve portability.
>
> Making the adjustor code more complicated by adding more functionality doesn’t sound like a good plan to me.
>
> Besides, there are other overheads in addition to the actual marshalling in FFI calls and most of the time we are calling out to library functions for which the FFI call overhead is only a small portion of the runtime.
 >
>>> i think the crux of Manuel's point is mainly that any good proposal
>>> has to
>>> at least give a roadmap to support on all the various platforms etc
>>> etc
>>
>> I don't think you are expecting detailed schedule from me. Passing
>> structure by value is possible on all platforms ghc supports, and it can
>> be implemented for any particular platform if somebody is interested.
>>
>>  From my point of view, at this point it is more important to agree on
>> the next question: do we want such functionality in ghc at all? I don't
>> want to waste time on it if nobody wants to see it merged.

I'm really keen to have support for returning structs in particular. 
Passing structs less so, because working around the lack of struct 
passing isn't nearly as onerous as working around the lack of struct 
returns.  Returning multiple values from a C function is a real pain 
without struct returns: you have to either allocate some memory in 
Haskell or in C, and both methods are needlessly complex and slow. 
(though allocating in Haskell is usually better.) C++ code does this all 
the time, so if you're wrapping C++ code for calling from Haskell, the 
lack of multiple returns bites a lot.

In fact implementing this is on my todo list, I'm really glad to see 
someone else is planning to do it :-)

The vague plan I had in my head was to allow the return value of a 
foreign import to be a tuple containing marshallable types, which would 
map to the appropriate return convention for a struct on the current 
platform.  Perhaps allowing it to be an arbitrary single-constructor 
type is better, because it allows us to use a type that has a Storable 
instance.

Cheers,
Simon

> I still don’t see the benefit in further complicating an already murky corner of the compiler. Moreover, for this to make sense, it would need to work on all supported platforms. Unless you are volunteering to implement it on multiple platforms, this would mean, we’d use libffi for most platforms anyway. This brings me to my original point, a library or tool is the better place for this.
>
> Manuel
>
> PS: I’d happily accept language-c-inline patches for marshalling structs.
>
>> On Sat, 2014-03-15 at 00:37 -0400, Carter Schonwald wrote:
>>> I'm not opposing that, in fact, theres a GHC ticket discussing some stuff
>>> related to this (related to complex numbers).
>>>
>>> i think the crux of Manuel's point is mainly that any good proposal has to
>>> at least give a roadmap to support on all the various platforms etc etc
>>>
>>>
>>> On Sat, Mar 15, 2014 at 12:33 AM, Edward Kmett <ekmett at gmail.com> wrote:
>>>
>>>> I don't care enough to fight and try to win the battle, but I just want to
>>>> point out that Storable structs are far more brittle and platform dependent
>>>> than borrowing the already correct platform logic for struct passing from
>>>> libffi.
>>>>
>>>> I do think the existing FFI extension made the right call under the 32 bit
>>>> ABIs that were in use at the time it was defined. That said, with 64-bit
>>>> ABIs saying that 2 32-bit ints should be passed in a single 64 bit
>>>> register, you wind up with large chunks of third party APIs we just can't
>>>> call out to directly any more, requiring many one-off manual C shims.
>>>>
>>>> -Edward
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Mar 15, 2014 at 12:17 AM, Carter Schonwald <
>>>> carter.schonwald at gmail.com> wrote:
>>>>
>>>>> indeed, its very very easy to do storable instances that correspond to
>>>>> the struct type you want,
>>>>>
>>>>> the ``with`` function in
>>>>> http://hackage.haskell.org/package/base-4.6.0.1/docs/Foreign-Marshal-Utils.htmlforeigh.marshal.utils actually gets you most of the way there!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Mar 15, 2014 at 12:00 AM, Manuel M T Chakravarty <
>>>>> chak at cse.unsw.edu.au> wrote:
>>>>>
>>>>>> Yuras,
>>>>>>
>>>>>> I’m not convinced that the compiler is the right place for this kind of
>>>>>> functionality. In fact, when we designed the Haskell FFI, we explicit
>>>>>> decided against what you propose. There are a few reasons for this.
>>>>>>
>>>>>> Firstly, compilers are complex beasts, and secondly, it takes a long
>>>>>> time until a change in the compiler goes into production. Hence, as a
>>>>>> general rule, it is advisable to move complexity from the compiler into
>>>>>> libraries as this reduces compiler complexity. Libraries are less complex
>>>>>> and changes can be rolled out much more quickly (it’s essentially a Hackage
>>>>>> upload versus waiting for the next GHC and Haskell Platform release).
>>>>>>
>>>>>> Thirdly, we have got the Haskell standard for a reason and modifying the
>>>>>> compiler implies a language extension.
>>>>>>
>>>>>> The design goal for the Haskell FFI was to provide the absolute minimum
>>>>>> as part of the language and compiler, and to layer additional conveniences
>>>>>> on top of that in the form of libraries and tools.
>>>>>>
>>>>>> Have you considered the library or tool route?
>>>>>>
>>>>>> Manuel
>>>>>>
>>>>>> Yuras Shumovich <shumovichy at gmail.com>:
>>>>>>> Hi,
>>>>>>>
>>>>>>> Right now ghc's FFI doesn't support c/c++ structures.
>>>>>>>
>>>>>>> Whenever we have foreign function that accepts or returns struct by
>>>>>>> value, we have to create wrapper that accepts or returns pointer to
>>>>>>> struct. It is inconvenient, but actually not a big deal.
>>>>>>>
>>>>>>> But there is no easy workaround when you want to export haskell
>>>>>> function
>>>>>>> to use it with c/c++ API that requires structures to be passed by value
>>>>>>> (Usually it is a callback in c/c++ API. You can't change it's
>>>>>> signature,
>>>>>>> and if it doesn't provide some kind of "void* userdata", then you are
>>>>>>> stuck.)
>>>>>>>
>>>>>>> I'm interested in fixing that. I'm going to start with 'foreign import
>>>>>>> "wrapper" ...' stuff.
>>>>>>>
>>>>>>> Calling conventions for passing c/c++ structures by value are pretty
>>>>>>> tricky and platform/compiler specific. So initially I'll use libffi for
>>>>>>> that (it will work when USE_LIBFFI_FOR_ADJUSTORS is defined, see
>>>>>>> rts/Adjustor.c). It will allow me to explore design space without
>>>>>>> bothering about low level implementation details. Later it could be
>>>>>>> implemented for native (non-libffi) adjustors.
>>>>>>>
>>>>>>> Is anybody interested it that? I appreciate any comments/ideas.
>>>>>>>
>>>>>>> Right now I don't have clear design. It would be nice to support plain
>>>>>>> haskell data types that are 1) not recursive, 2) has one constructor
>>>>>> and
>>>>>>> 3) contains only c/c++ types. But it doesn't work with c/c++ unions.
>>>>>> Any
>>>>>>> ideas are welcome.
>>>>>>>
>>>>>>> An example how to use libffi with structures:
>>>>>>> http://www.atmark-techno.com/~yashi/libffi.html#Structures
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Yuras
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> ghc-devs mailing list
>>>>>>> ghc-devs at haskell.org
>>>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>>>>>
>>>>>> _______________________________________________
>>>>>> ghc-devs mailing list
>>>>>> ghc-devs at haskell.org
>>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> ghc-devs mailing list
>>>>> ghc-devs at haskell.org
>>>>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://www.haskell.org/mailman/listinfo/ghc-devs
>>
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://www.haskell.org/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>