[Msys2-users] Debugging undeterministic segfaults

Tue Nov 4 00:45:09 UTC 2014

I have a suspicion that the segmentation faults are related to fork errors,
which appear in logs from time to time. After a few builds fork() errors
become very frequent, and segfaults seem to occur more often. I just ran
the build (validate.sh) in a loop, and after a while got a bunch of
segfaults in Makefile rules as basic as rm invocations in the initial "make
clean", e.g.:

"rm" -rf driver/split/dist
driver/split/ghc.mk:19: recipe for target 'clean_driver/split_dist' failed
make[1]: *** [clean_driver/split_dist] Segmentation fault
Makefile:94: recipe for target 'maintainer-clean' failed

Closing and reopening the msys2 console seems to help - for some time.

On Tue, Nov 4, 2014 at 1:30 AM, Ray Donnelly <mingw.android at gmail.com>
wrote:

> On Mon, Nov 3, 2014 at 11:07 PM, Gintautas Miliauskas
> <gintautas.miliauskas at gmail.com> wrote:
> > +ghc-devs
> >
> > Hi Ray,
> >
> > thanks for the feedback. ghc-stage1 is a native application, it is built
> > using a host ghc and a mingw gcc bundled with the ghc distribution (in
> order
> > to keep Windows builds more predictable). The thing is, the same builds
> > (using make) that were stable on cygwin seem to be crashing on msys2,
> even
> > though the (bundled) gcc used for the build is the same. It could be that
> > msys2 is triggering a bug in ghc somehow, but it could be something
> going on
> > in msys2 itself.
> >
> > The tricky part is that the crashes are rare, one in thousands of ghc
> > invocations within a make build, and I'm having trouble pinning one down
> to
> > investigate. I'll try some basic tracing first, but ideas for something
> more
> > sophisticated would be very helpful.
>
> Ah, ok. I skim read your initial email and applied totally incorrect
> weighting to the "not very hard to reproduce" bit, apologies!
>
> It *should* be possible to setup post-mortem debugging using either Qt
> Creator (Tools->Options->Debugger, tick "Use Qt Creator for
> post-mortem debugging") or Dr. Mingw
> (https://github.com/jrfonseca/drmingw). I briefly tested both options:
>
> Qt Creator: It seems there's a bug in its handling of the
> -wincrashevent command line which I've just added a note about to the
> MINGW-packages todo list since I'd find this feature more than very
> useful:
> https://github.com/Alexpux/MINGW-packages/blob/master/TODO.md
>
> Dr Mingw: Launching the crashing executable from Windows Explorer, it
> launches it and gives me useful information.
>
> Unfortunately, regardless of the debugger used, when invoking the
> crashing executable from the mintty commandline, something inhibits
> all post mortem debugging. I will ask the mingw-w64 guys if they know
> what this is. I guess intrusive dialogs would be annoying, but I'd
> like an env. var override for that I think.
>
> There is another possibility, and that's that bash is crashing, which
> is an MSYS2 program. We do have a way of hooking into post-mortem
> debugging there since Cygwin provided a way and we improved it. If you
> check your msys2_shell.bat you'll see:
> rem set
> MSYS=error_start:%WD%../../mingw64/bin/qtcreator.exe|-debug|<process-id>
>
> If you remove the rem and make sure the path is correct (it may have
> rotted some since I last used it) then hopefully you'll get something
> useful. To be really useful, you should rebuild two packages,
> MSYS2-packages/{msys2-runtime,bash}/PKGBUILD with
> options=('debug' '!strip')
> and then install them.
>
> Finally, can anyone else confirm the problem?
>
> There may be some edge cases when e.g. PATH is around 1024 characters,
> I've seen some hardcoded limits in the msys2-runtime (nee Cygwin)
> code base for things like that. I'd advise cutting your Windows PATH
> down to just the essentials.
>
> Cheers,
>
> Ray.
>
> >
> > A PKGBUILD for ghc should be feasible, although the bootstrapping is a
> bit
> > tricky (some Haskell tools need to be preinstalled). I'm not sure how
> useful
> > it would be since for Windows there is already a nicely packaged native
> > Haskell Platform installer.
> >
> >
> > On Mon, Nov 3, 2014 at 3:30 AM, Ray Donnelly <mingw.android at gmail.com>
> > wrote:
> >>
> >> On Sun, Nov 2, 2014 at 11:45 PM, Gintautas Miliauskas
> >> <gintautas.miliauskas at gmail.com> wrote:
> >> > Hello,
> >> >
> >> > I have been working on building GHC, the Glasgow Haskell Compiler, on
> >> > Windows using msys2 [1]. We have been having some strange trouble with
> >> > ghc
> >> > segfaulting during the bootstrapping process (i.e., during make). The
> >> > crashes are not very hard to reproduce, but they are not predictable
> >> > either.
> >> > This is almost certainly a bug in ghc itself and not with msys2
> >> > (although
> >> > the crashes do not seem to occur in other environments), but I've been
> >> > having some trouble pinning it down.
> >> >
> >>
> >> Hi Gintuatas,
> >>
> >> Great. I spotted that MSYS2 was the recommended env. for GHC on
> >> Windows, hopefully it will remain so!
> >>
> >> > What's a good way to debug such crashes? Is it possible to somehow
> stop
> >> > the
> >> > ghc process when it segfaults and attach gdb, or to dump core somehow
> >> > for
> >> > later inspection?
> >> >
> >> > [1]
> https://ghc.haskell.org/trac/ghc/wiki/Building/Preparation/Windows
> >> >
> >> > Here's one example crash:
> >> >
> >> > "inplace/bin/ghc-stage1.exe" -hisuf hi -osuf  o -hcsuf hc -static
> -H32m
> >> > -O
> >> > -Werror -Wall -H64m -O0    -this-package-key
> ghc_4ugNArSu5ba0Z1uHXrbTlU
> >> > -hide-all-packages -i -icompiler/basicTypes -icompiler/cmm
> >> > -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar
> -icompiler/ghci
> >> > -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen -icompiler/main
> >> > -icompiler/nativeGen -icompiler/parser -icompiler/prelude
> >> > -icompiler/profiling -icompiler/rename -icompiler/simplCore
> >> > -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn
> >> > -icompiler/stranal -icompiler/typecheck -icompiler/types
> >> > -icompiler/utils
> >> > -icompiler/vectorise -icompiler/stage2/build
> >> > -icompiler/stage2/build/autogen
> >> > -Icompiler/stage2/build -Icompiler/stage2/build/autogen -Icompiler/.
> >> > -Icompiler/parser -Icompiler/utils -Icompiler/../rts/dist/build
> >> > -Icompiler/stage2   -optP-DGHCI -optP-include
> >> > -optPcompiler/stage2/build/autogen/cabal_macros.h -package-key
> >> > Win32_43THQMouBnk2wpnouztX1X -package-key array_GX4NwjS8xZkC2ZPtjgwhnz
> >> > -package-key base_ESD4aQEEWwsHtYJVc1BwtJ -package-key
> >> > binpa_17GphrLqCXt1H1lm4Kse1p -package-key bytes_Kc0PyaputnzDnBdZW0y2Gv
> >> > -package-key conta_ChF4XLXB9JmByIycPzerow -package-key
> >> > direc_HU5aFxMIQNwGQFzisjuinu -package-key filep_34DFDFT9FVD9pRLLgh8IdQ
> >> > -package-key hoopl_IZAd44CED5NCOlpg8p2Kaj -package-key
> >> > hpc_1QTsfQSN40FHN9p3mydARY -package-key proce_7ZlAbRkwiRO8qgXx3NNP0G
> >> > -package-key templ_F6UJgDtBcDIFWuHmMGEFzy -package-key
> >> > time_HGs4JcQCd4wF6U8vJQ5fNH -package-key trans_5jw4w9yTgmZ89ByuixDAKP
> >> > -Wall
> >> > -fno-warn-name-shadowing -this-package-key ghc -XHaskell2010
> >> > -optc-DTHREADED_RTS -DGHCI_TABLES_NEXT_TO_CODE -DSTAGE=2 -Rghc-timing
> >> > -O2
> >> > -fwarn-tabs -O -dcore-lint  -no-user-package-db -rtsopts      -odir
> >> > compiler/stage2/build -hidir compiler/stage2/build -stubdir
> >> > compiler/stage2/build   -c compiler/basicTypes/UniqSupply.lhs -o
> >> > compiler/stage2/build/UniqSupply.o
> >> >
> >> > compiler/ghc.mk:657: recipe for target
> >> > 'compiler/stage2/build/UniqSupply.o'
> >> > failed
> >> > make[1]: *** [compiler/stage2/build/UniqSupply.o] Segmentation fault
> >> >
> >>
> >> Well, it's pretty much the same story as with Linux. Build the
> >> executable and as many libraries as you can with (something like)
> >> "-ggdb -O0" then use gdb command line or some IDE of choice.
> >> Personally I use Qt Creator (we have packages for this) as it can
> >> debug external programs and, even though I don't dislike command
> >> lines, I find debugging from them a bit too masochistic.
> >>
> >> Is ghc-stage1.exe an MSYS2 program or a native one? Which compiler is
> >> it built with exactly?
> >>
> >> What would be involved in creating a PKGBUILD for ghc? We'd love to
> >> have one, and it'd certainly make me a lot more inclined to dive in to
> >> see what's going on if one existed already.
> >>
> >> Ray.
> >>
> >> >
> >> > --
> >> > Gintautas Miliauskas
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> >
> >> > _______________________________________________
> >> > Msys2-users mailing list
> >> > Msys2-users at lists.sourceforge.net
> >> > https://lists.sourceforge.net/lists/listinfo/msys2-users
> >> >
> >
> >
> >
> >
> > --
> > Gintautas Miliauskas
>

-- 
Gintautas Miliauskas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20141104/447f70f8/attachment-0001.html>