Windows test failures

[Phyx]

Hi Roland,

Thanks for looking into thiis,

> To fix the issue on Windows, the compiler and the plugin should use the
same buffer for stdout.

I'm not convinced that they're not. I'm guessing the answer lies in why
your messages have a different
ordering, but I don't know why off the top of my head.

if the plugins use the stdout caf then it should be using the same
CharBuffer.

You can verify this by doing something like

cbuf <- readIORef haCharBuffer stdout
summaryBuffer cbuf
putStrLn $ "buffer: " ++ show cbuf

at the places you check the buffer mode.

> However I don't know whether this is possible / difficult / easy?
> What's your opinion?

On Tue, Dec 4, 2018 at 11:52 AM Roland Senn <rsx at bluewin.ch> wrote:

> Hi Tamar,
>
> WINDOWS
> =======
> On Windows I did the following changes before running the 'plugin09' test:
> 1.) In the compiler (HscMain.hs), just before calling the plugin function
> 'parsedPlugin', I set BufferMode of the file stdout to LineBuffering.
> 2.) At the same place, I write a message to stdout with the text "COMPILER
> About to call plugin parse" and the result of the buffer-mode query.
> 3.) In the plugin, in the function parsedPlugin, I query and print (to
> stdout) the buffer mode.
> 4.) I added the heading PLUGIN to the normal parse message issued by the
> parsedPlugin function
> 5.) In the compiler (HscMain.hs) just after returning from the plugin, I
> print the line "COMPILER Returning from plugin parse" to stdout.
> 6.) In the  plugin function interfaceLoadPlugin' that is called much
> later, I flush stdout, and add the heading "PLUGIN".
>
> This gives the following interesting result:
>
>  COMPILER About to call plugin parse: LineBuffering
>  COMPILER Returning from plugin parse
>  PLUGIN Buffermode: BlockBuffering Nothing
>  PLUGIN parsePlugin(a,b)
>  PLUGIN interfacePlugin: Prelude
>  ...
>
> The output lines do not appear in the sequence they were produced!!
> The plugin doesn't see/inherit the BlockBuffer mode (LineBuffering) set by
> the compiler!!
>
> This is a strong indication, that *there are two different buffers for
> stdout*. One in the compiler and another one in the plugin.
> At the end of the processing, the buffer in the compiler is automatically
> flushed, however the buffer in the plugin never gets flushed!
>
> LINUX
> =====
> I did a similar test in Linux, however, here I set the buffer mode to
> 'Blockmode Nothing' and I didn't do a manual flush in the plugin. I got the
> following result:
>
>  COMPILER About to call plugin parse: Buffering mode: BlockBuffering
> Nothing
>  PLUGIN Buffering: BlockBuffering Nothing
>  PLUGIN parsePlugin(a,b)
>  COMPILER Returning from plugin parse
>  PLUGIN interfacePlugin: Prelude
>  ...
>
> Here the lines are in the same order as they were produced.
> The setting of the Buffering mode is inherited by the plugin.
>
> I think, on Linux the compiler and the plugin share the same buffer.
>
> To fix the issue on Windows, the compiler and the plugin should use the
> same buffer for stdout.
> However I don't know whether this is possible / difficult / easy?
> What's your opinion?
>
> Many thanks and kind regards
>    Roland
>
> Here are my changes for Windows in code:
>
> Change in HscMain the line "import System.IO (fixIO)" to "import System.IO
> "
>
> Last lines of function HscMain.hs:hscParse'
>
>             -- apply parse transformation of plugins
>             let applyPluginAction p opts
>                   = parsedResultAction p opts mod_summary
>             liftIO $ hSetBuffering stdout LineBuffering
>             mode <- liftIO $ hGetBuffering stdout
>             liftIO $ putStrLn ("COMPILER About to call plugin parse: " ++
> show mode)
>             rsxresult <- withPlugins dflags applyPluginAction res
>             liftIO $ putStrLn "COMPILER Returning from plugin parse"
>             return rsxresult
>
> New code for function SourcePlugin.hs:parsedPlugin
>
> parsedPlugin opts _ pm
>   = do
>        mode <- liftIO $ hGetBuffering stdout
>        liftIO $ putStrLn $ "PLUGIN Buffermode: " ++ show mode
>        liftIO $ putStrLn $ "PLUGIN parsePlugin(" ++ intercalate "," opts
> ++ ")"
>        return pm
>
> New code for function SourcePlugin.hs:interfaceLoadPlugin'
>
> interfaceLoadPlugin' :: [CommandLineOption] -> ModIface -> IfM lcl ModIface
> interfaceLoadPlugin' _ iface
>   = do liftIO $ putStrLn $ "PLUGIN interfacePlugin: "
>                               ++ (showSDocUnsafe $ ppr $ mi_module iface)
>        liftIO $ hFlush stdout
>        return iface
>
>
>
> Am Dienstag, den 04.12.2018, 00:02 +0000 schrieb Phyx:
>
> Hi Roland,
>
> Thanks for looking into these.
>
> > I looked into the testcases 'plugins09', 'plugins10' and 'plugins11' and
> found the following: GHC-Windows uses BufferMode 'BlockBuffering Nothing',
> however, GHC-Linux uses 'LineBuffering'.
>
> Ah, yes, this isn't technically a Linux vs Windows thing, GHC will always
> default to LineBuffering for terminals and BlockBuffering for anything
> else. The issue is that msys2 terminals have std handles that are backed by
> files instead of pipes. This is an artifact of how they try to emulate
> posix shells, See
> https://github.com/msys2/msys2/wiki/Porting#standard-streams-in-mintty.
> This means that when GHC runs inside an msys2 program such as bash it'll
> always default to BlockBuffering.
>
> > I don't know anything about the "Why" and "Where" in the GHC IO module
> on Windows, so I'm unable to come up with a patch.
>
> The above said the handles should be getting flushed when the finalizers
> are run, but these aren't 100% guaranteed so if the tests rely on this then
> your solution (to force buffer mode to LineBuffering) sounds like perfectly
> adequate.  Could you put a patch up with that?
>
> My new I/O manager takes a different approach to determine the buffer
> mode, but I still have some kinks to work out before posting it.
>
> > PS: I can't say anything about the tests 'plugin-recomp-pure' and
> 'plugin-recomp-impure' as these tests run successfully on my (slow) Windows
> box.
>
> These don't fail for me or harbormaster either, so if Simon is able to
> consistently reproduce these then I'll have to ask him for a core dump so I
> can take a look.
>
> Thanks again,
> I appreciate the help!
>
> Regards,
> Tamar
>
> On Mon, Dec 3, 2018 at 3:34 PM Roland Senn <rsx at bluewin.ch> wrote:
>
> Hi Tamar,
>
> I looked into the testcases 'plugins09', 'plugins10' and 'plugins11' and
> found the following: GHC-Windows uses BufferMode 'BlockBuffering Nothing',
> however, GHC-Linux uses 'LineBuffering'.
>
> If I add an 'import System.IO' and the line 'liftIO $ hSetBuffering stdout
> LineBuffer' as first line in the do block of the function
> '.../testuite/tests/plugins/simple-plugin/Simple/SourcePlugin.hs:parsedPlugin'
> then all 3 tests pass successfully on Windows!
>
> I don't know anything about the "Why" and "Where" in the GHC IO module on
> Windows, so I'm unable to come up with a patch.
>
> Regards
>    Roland
>
> PS: I can't say anything about the tests 'plugin-recomp-pure' and
> 'plugin-recomp-impure' as these tests run successfully on my (slow) Windows
> box.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20181206/d49b8392/attachment.html>