[GHC] #8591: Concurrent executions of ghc-pkg can cause inconstant package.cache files

[GHC]

#8591: Concurrent executions of ghc-pkg can cause inconstant package.cache files
------------------------------------+----------------------------
       Reporter:  janm              |             Owner:
           Type:  bug               |            Status:  new
       Priority:  normal            |         Milestone:
      Component:  Package system    |           Version:  7.6.3
       Keywords:  ghc-pkg race      |  Operating System:  FreeBSD
   Architecture:  Unknown/Multiple  |   Type of failure:  Other
     Difficulty:  Unknown           |         Test Case:
     Blocked By:                    |          Blocking:
Related Tickets:                    |
------------------------------------+----------------------------
 I am doing 24 way parallel builds of system images, including all packages
 on a system. This includes ghc and multiple ghc packages.

 I am seeing intermittent dependency failure from the ghc packaging system.
 From examining Main.hs in ghc-pkg, I see the function withFileAtomic write
 to a temporary file in package.conf.d and then atomically rename on top of
 a target file, package.cache in the case. With parallel execution the last
 rename would win, leading to lost entries in package.cache.

 In my case, the following things happened: ("Building" indicates a start,
 "Built" indicates completion, "Installing" is setup in a separate chroot'd
 environment and is isolated)

 The FreeBSD ports system is used to drive the Haskell build system.

 The process works single threaded and fails intermittently when done in
 parallel.

 Building: devel/hs-data-default-instances-base
 Building: devel/hs-data-default-instances-containers
 Building: devel/hs-data-default-instances-old-locale
 Built: devel/hs-dlist
 Building: devel/hs-data-default-instances-dlist
 Built: devel/hs-temporary
 Built: jail-image-full
 Installing: system-image__jail-image-full
 Built: devel/hs-base64-bytestring
 Built: archivers/hs-zlib
 Building: security/hs-digest
 Built: devel/hs-syb
 Building: textproc/hs-hs-bibutils
 Building: textproc/hs-pandoc-types
 Built: devel/hs-utf8-string
 Built: devel/hs-data-default-instances-old-locale
 Built: devel/hs-data-default-instances-containers
 Built: devel/hs-data-default-instances-base
 Built: devel/hs-data-default-instances-dlist
 Building: devel/hs-data-default
 Built: devel/hs-random
 Installed: system-image__lang/ghc
 Installing: system-image__archivers/hs-zlib
 Installing: system-image__devel/hs-utf8-string
 Installing: system-image__devel/hs-syb
 Installing: system-image__devel/hs-base64-bytestring
 Installing: system-image__devel/hs-data-default-class
 Installing: system-image__devel/hs-dlist
 Installing: system-image__devel/hs-random
 Installing: system-image__devel/hs-temporary
 Installing: system-image__devel/hs-extensible-exceptions
 Built: devel/hs-data-default FAILED

 The error from the Haskell data-default build was:

 setup: At least the following dependencies are missing:
 data-default-instances-base -any

 Looking in the in the package.conf.d directory shows that the data-
 default-instances-base-0.0.1-7bdf8678f0d8637e096e397e7910f82a.conf file
 was present, but running "ghc-pkg list" did not show data-default-
 instances-base

 Running /usr/local/lib/cabal/ghc-7.6.3/data-default-instances-
 base-0.0.1/register.sh (which was also present) caused ghc-pkg to now show
 data-default-instances-base.

 To me this looks like a race condition between multiple instances of ghc-
 pkg causing the cache to become inconsistent.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/8591>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler