[Msys2-users] Debugging undeterministic segfaults
Gintautas Miliauskas
gintautas.miliauskas at gmail.com
Tue Nov 4 12:07:49 UTC 2014
I had a problem <https://github.com/Alexpux/MSYS2-packages/issues/116> with
building msys2-runtime, but debug bash installed successfully.
I uncommented the hook to invoke qtcreator.exe on crashing (your .bat file
needs escaping, puill request pending...), but I'm not quite sure how to
check that it works. I tried building this crasher program with
/usr/bin/gcc:
#include <stdio.h>
int main() {
char *s = 0;
printf("hi\n");
printf("%s\n", s);
printf("bye\n");
return 0;
}
But running it just prints "hi", no indications of a crash or anything,
regardless of the value of the MSYS variable:
$ /usr/bin/gcc -o crash crash.c
$ ./crash.exe
hi
$ echo $?
0
On Tue, Nov 4, 2014 at 1:56 AM, Ray Donnelly <mingw.android at gmail.com>
wrote:
> On Tue, Nov 4, 2014 at 12:45 AM, Gintautas Miliauskas
> <gintautas.miliauskas at gmail.com> wrote:
> > I have a suspicion that the segmentation faults are related to fork
> errors,
> > which appear in logs from time to time. After a few builds fork() errors
> > become very frequent, and segfaults seem to occur more often. I just ran
> the
> > build (validate.sh) in a loop, and after a while got a bunch of
> segfaults in
> > Makefile rules as basic as rm invocations in the initial "make clean",
> e.g.:
> >
> > "rm" -rf driver/split/dist
> > driver/split/ghc.mk:19: recipe for target 'clean_driver/split_dist'
> failed
> > make[1]: *** [clean_driver/split_dist] Segmentation fault
> > Makefile:94: recipe for target 'maintainer-clean' failed
> >
> > Closing and reopening the msys2 console seems to help - for some time.
>
> Interesting .. how much process spawning do you think happens? On
> MSYS2 we run massive builds (bootstrapping builds of GCC for 2 arches
> for example), but MSYS2 shares a memory map, so heap corruption could
> happen and cause later problems. My recommendation about building
> debug bash and msys2-runtime and set
> MSYS=error_start:%WD%../../mingw64/bin/qtcreator.exe|-debug|<process-id>
> stands all the more then. You may need to setup path mappings in Qt C
> to map from MSYS2 paths to Windows paths. You need to do this as QtC
> needs to use the native GDB instead of the MSYS2 one so that QtC and
> GDB are in the same pathing namespace.
>
> There may be a way of building msys2-runtime to check for memory
> overruns too, perhaps just building it with the 'debug' option will
> enable that, I'm not sure ..
>
> >
> >
> >
> >
> > On Tue, Nov 4, 2014 at 1:30 AM, Ray Donnelly <mingw.android at gmail.com>
> > wrote:
> >>
> >> On Mon, Nov 3, 2014 at 11:07 PM, Gintautas Miliauskas
> >> <gintautas.miliauskas at gmail.com> wrote:
> >> > +ghc-devs
> >> >
> >> > Hi Ray,
> >> >
> >> > thanks for the feedback. ghc-stage1 is a native application, it is
> built
> >> > using a host ghc and a mingw gcc bundled with the ghc distribution (in
> >> > order
> >> > to keep Windows builds more predictable). The thing is, the same
> builds
> >> > (using make) that were stable on cygwin seem to be crashing on msys2,
> >> > even
> >> > though the (bundled) gcc used for the build is the same. It could be
> >> > that
> >> > msys2 is triggering a bug in ghc somehow, but it could be something
> >> > going on
> >> > in msys2 itself.
> >> >
> >> > The tricky part is that the crashes are rare, one in thousands of ghc
> >> > invocations within a make build, and I'm having trouble pinning one
> down
> >> > to
> >> > investigate. I'll try some basic tracing first, but ideas for
> something
> >> > more
> >> > sophisticated would be very helpful.
> >>
> >> Ah, ok. I skim read your initial email and applied totally incorrect
> >> weighting to the "not very hard to reproduce" bit, apologies!
> >>
> >> It *should* be possible to setup post-mortem debugging using either Qt
> >> Creator (Tools->Options->Debugger, tick "Use Qt Creator for
> >> post-mortem debugging") or Dr. Mingw
> >> (https://github.com/jrfonseca/drmingw). I briefly tested both options:
> >>
> >> Qt Creator: It seems there's a bug in its handling of the
> >> -wincrashevent command line which I've just added a note about to the
> >> MINGW-packages todo list since I'd find this feature more than very
> >> useful:
> >> https://github.com/Alexpux/MINGW-packages/blob/master/TODO.md
> >>
> >> Dr Mingw: Launching the crashing executable from Windows Explorer, it
> >> launches it and gives me useful information.
> >>
> >> Unfortunately, regardless of the debugger used, when invoking the
> >> crashing executable from the mintty commandline, something inhibits
> >> all post mortem debugging. I will ask the mingw-w64 guys if they know
> >> what this is. I guess intrusive dialogs would be annoying, but I'd
> >> like an env. var override for that I think.
> >>
> >> There is another possibility, and that's that bash is crashing, which
> >> is an MSYS2 program. We do have a way of hooking into post-mortem
> >> debugging there since Cygwin provided a way and we improved it. If you
> >> check your msys2_shell.bat you'll see:
> >> rem set
> >> MSYS=error_start:%WD%../../mingw64/bin/qtcreator.exe|-debug|<process-id>
> >>
> >> If you remove the rem and make sure the path is correct (it may have
> >> rotted some since I last used it) then hopefully you'll get something
> >> useful. To be really useful, you should rebuild two packages,
> >> MSYS2-packages/{msys2-runtime,bash}/PKGBUILD with
> >> options=('debug' '!strip')
> >> and then install them.
> >>
> >> Finally, can anyone else confirm the problem?
> >>
> >> There may be some edge cases when e.g. PATH is around 1024 characters,
> >> I've seen some hardcoded limits in the msys2-runtime (nee Cygwin)
> >> code base for things like that. I'd advise cutting your Windows PATH
> >> down to just the essentials.
> >>
> >> Cheers,
> >>
> >> Ray.
> >>
> >> >
> >> > A PKGBUILD for ghc should be feasible, although the bootstrapping is a
> >> > bit
> >> > tricky (some Haskell tools need to be preinstalled). I'm not sure how
> >> > useful
> >> > it would be since for Windows there is already a nicely packaged
> native
> >> > Haskell Platform installer.
> >> >
> >> >
> >> > On Mon, Nov 3, 2014 at 3:30 AM, Ray Donnelly <mingw.android at gmail.com
> >
> >> > wrote:
> >> >>
> >> >> On Sun, Nov 2, 2014 at 11:45 PM, Gintautas Miliauskas
> >> >> <gintautas.miliauskas at gmail.com> wrote:
> >> >> > Hello,
> >> >> >
> >> >> > I have been working on building GHC, the Glasgow Haskell Compiler,
> on
> >> >> > Windows using msys2 [1]. We have been having some strange trouble
> >> >> > with
> >> >> > ghc
> >> >> > segfaulting during the bootstrapping process (i.e., during make).
> The
> >> >> > crashes are not very hard to reproduce, but they are not
> predictable
> >> >> > either.
> >> >> > This is almost certainly a bug in ghc itself and not with msys2
> >> >> > (although
> >> >> > the crashes do not seem to occur in other environments), but I've
> >> >> > been
> >> >> > having some trouble pinning it down.
> >> >> >
> >> >>
> >> >> Hi Gintuatas,
> >> >>
> >> >> Great. I spotted that MSYS2 was the recommended env. for GHC on
> >> >> Windows, hopefully it will remain so!
> >> >>
> >> >> > What's a good way to debug such crashes? Is it possible to somehow
> >> >> > stop
> >> >> > the
> >> >> > ghc process when it segfaults and attach gdb, or to dump core
> somehow
> >> >> > for
> >> >> > later inspection?
> >> >> >
> >> >> > [1]
> >> >> > https://ghc.haskell.org/trac/ghc/wiki/Building/Preparation/Windows
> >> >> >
> >> >> > Here's one example crash:
> >> >> >
> >> >> > "inplace/bin/ghc-stage1.exe" -hisuf hi -osuf o -hcsuf hc -static
> >> >> > -H32m
> >> >> > -O
> >> >> > -Werror -Wall -H64m -O0 -this-package-key
> >> >> > ghc_4ugNArSu5ba0Z1uHXrbTlU
> >> >> > -hide-all-packages -i -icompiler/basicTypes -icompiler/cmm
> >> >> > -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar
> >> >> > -icompiler/ghci
> >> >> > -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen
> -icompiler/main
> >> >> > -icompiler/nativeGen -icompiler/parser -icompiler/prelude
> >> >> > -icompiler/profiling -icompiler/rename -icompiler/simplCore
> >> >> > -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn
> >> >> > -icompiler/stranal -icompiler/typecheck -icompiler/types
> >> >> > -icompiler/utils
> >> >> > -icompiler/vectorise -icompiler/stage2/build
> >> >> > -icompiler/stage2/build/autogen
> >> >> > -Icompiler/stage2/build -Icompiler/stage2/build/autogen
> -Icompiler/.
> >> >> > -Icompiler/parser -Icompiler/utils -Icompiler/../rts/dist/build
> >> >> > -Icompiler/stage2 -optP-DGHCI -optP-include
> >> >> > -optPcompiler/stage2/build/autogen/cabal_macros.h -package-key
> >> >> > Win32_43THQMouBnk2wpnouztX1X -package-key
> >> >> > array_GX4NwjS8xZkC2ZPtjgwhnz
> >> >> > -package-key base_ESD4aQEEWwsHtYJVc1BwtJ -package-key
> >> >> > binpa_17GphrLqCXt1H1lm4Kse1p -package-key
> >> >> > bytes_Kc0PyaputnzDnBdZW0y2Gv
> >> >> > -package-key conta_ChF4XLXB9JmByIycPzerow -package-key
> >> >> > direc_HU5aFxMIQNwGQFzisjuinu -package-key
> >> >> > filep_34DFDFT9FVD9pRLLgh8IdQ
> >> >> > -package-key hoopl_IZAd44CED5NCOlpg8p2Kaj -package-key
> >> >> > hpc_1QTsfQSN40FHN9p3mydARY -package-key
> proce_7ZlAbRkwiRO8qgXx3NNP0G
> >> >> > -package-key templ_F6UJgDtBcDIFWuHmMGEFzy -package-key
> >> >> > time_HGs4JcQCd4wF6U8vJQ5fNH -package-key
> trans_5jw4w9yTgmZ89ByuixDAKP
> >> >> > -Wall
> >> >> > -fno-warn-name-shadowing -this-package-key ghc -XHaskell2010
> >> >> > -optc-DTHREADED_RTS -DGHCI_TABLES_NEXT_TO_CODE -DSTAGE=2
> -Rghc-timing
> >> >> > -O2
> >> >> > -fwarn-tabs -O -dcore-lint -no-user-package-db -rtsopts -odir
> >> >> > compiler/stage2/build -hidir compiler/stage2/build -stubdir
> >> >> > compiler/stage2/build -c compiler/basicTypes/UniqSupply.lhs -o
> >> >> > compiler/stage2/build/UniqSupply.o
> >> >> >
> >> >> > compiler/ghc.mk:657: recipe for target
> >> >> > 'compiler/stage2/build/UniqSupply.o'
> >> >> > failed
> >> >> > make[1]: *** [compiler/stage2/build/UniqSupply.o] Segmentation
> fault
> >> >> >
> >> >>
> >> >> Well, it's pretty much the same story as with Linux. Build the
> >> >> executable and as many libraries as you can with (something like)
> >> >> "-ggdb -O0" then use gdb command line or some IDE of choice.
> >> >> Personally I use Qt Creator (we have packages for this) as it can
> >> >> debug external programs and, even though I don't dislike command
> >> >> lines, I find debugging from them a bit too masochistic.
> >> >>
> >> >> Is ghc-stage1.exe an MSYS2 program or a native one? Which compiler is
> >> >> it built with exactly?
> >> >>
> >> >> What would be involved in creating a PKGBUILD for ghc? We'd love to
> >> >> have one, and it'd certainly make me a lot more inclined to dive in
> to
> >> >> see what's going on if one existed already.
> >> >>
> >> >> Ray.
> >> >>
> >> >> >
> >> >> > --
> >> >> > Gintautas Miliauskas
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> ------------------------------------------------------------------------------
> >> >> >
> >> >> > _______________________________________________
> >> >> > Msys2-users mailing list
> >> >> > Msys2-users at lists.sourceforge.net
> >> >> > https://lists.sourceforge.net/lists/listinfo/msys2-users
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Gintautas Miliauskas
> >
> >
> >
> >
> > --
> > Gintautas Miliauskas
>
--
Gintautas Miliauskas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20141104/40a0330f/attachment-0001.html>
More information about the ghc-devs
mailing list