[Git][ghc/ghc][wip/marge_bot_batch_merge_job] 8 commits: Use setSrcSpan rather than setLclEnv in solveForAll
Marge Bot (@marge-bot)
gitlab at gitlab.haskell.org
Thu May 18 09:47:43 UTC 2023
Marge Bot pushed to branch wip/marge_bot_batch_merge_job at Glasgow Haskell Compiler / GHC
Commits:
5fe1d3e6 by Matthew Pickering at 2023-05-17T21:42:00-04:00
Use setSrcSpan rather than setLclEnv in solveForAll
In subsequent MRs (#23409) we want to remove the TcLclEnv argument from
a CtLoc. This MR prepares us for that by removing the one place where
the entire TcLclEnv is used, by using it more precisely to just set the
contexts source location.
Fixes #23390
- - - - -
385edb65 by Torsten Schmits at 2023-05-17T21:42:40-04:00
Update the users guide paragraph on -O in GHCi
In relation to #23056
- - - - -
d0650886 by Ben Gamari at 2023-05-18T05:47:24-04:00
base: Add test for #13660
- - - - -
3a972838 by Ben Gamari at 2023-05-18T05:47:24-04:00
base: Move implementation of GHC.Foreign to GHC.Internal
- - - - -
627446cb by Ben Gamari at 2023-05-18T05:47:24-04:00
base: Introduce {new,with}CStringLen0
These are useful helpers for implementing the internal-NUL code unit
check needed to fix #13660.
- - - - -
fdd0582d by Ben Gamari at 2023-05-18T05:47:24-04:00
base: Clean up documentation
- - - - -
1a9322ee by Ben Gamari at 2023-05-18T05:47:24-04:00
base: Ensure that FilePaths don't contain NULs
POSIX filepaths may not contain the NUL octet but previously we did not
reject such paths. This could be exploited by untrusted input to cause
discrepancies between various `FilePath` queries and the opened
filename. For instance, `readFile "hello.so\x00.txt"` would open the
file `"hello.so"` yet `takeFileExtension` would return `".txt"`.
The same argument applies to Windows FilePaths
Fixes #13660.
- - - - -
70c6bca2 by Torsten Schmits at 2023-05-18T05:47:26-04:00
Update the warning about interpreter optimizations
to reflect that they're not incompatible anymore, but guarded by a flag
- - - - -
17 changed files:
- compiler/GHC/Driver/Session.hs
- compiler/GHC/Tc/Solver/Canonical.hs
- compiler/GHC/Tc/Solver/Monad.hs
- docs/users_guide/ghci.rst
- libraries/base/GHC/Foreign.hs
- + libraries/base/GHC/Foreign/Internal.hs
- libraries/base/System/Posix/Internals.hs
- libraries/base/base.cabal
- + libraries/base/tests/T13660.hs
- + libraries/base/tests/T13660.stdout
- libraries/base/tests/all.T
- testsuite/tests/ghc-api/T10052/T10052.stderr
- testsuite/tests/ghci.debugger/scripts/print007.stderr
- testsuite/tests/ghci/should_fail/T10549.stderr
- testsuite/tests/ghci/should_fail/T10549a.stderr
- testsuite/tests/safeHaskell/ghci/p14.stderr
- testsuite/tests/th/T8333.stderr
Changes:
=====================================
compiler/GHC/Driver/Session.hs
=====================================
@@ -3658,9 +3658,10 @@ makeDynFlagsConsistent dflags
, gopt Opt_UnoptimizedCoreForInterpreter dflags
, let (dflags', changed) = updOptLevelChanged 0 dflags
, changed
- = loop dflags' ("Optimization flags are incompatible with the " ++
- backendDescription (backend dflags) ++
- "; optimization flags ignored.")
+ = loop dflags' $
+ "Using optimization flags with the " ++
+ backendDescription (backend dflags) ++ " is experimental. " ++
+ "Pass -fno-unoptimized-core-for-interpreter to enable this feature."
| LinkInMemory <- ghcLink dflags
, not (gopt Opt_ExternalInterpreter dflags)
=====================================
compiler/GHC/Tc/Solver/Canonical.hs
=====================================
@@ -53,6 +53,7 @@ import GHC.Data.Bag
import Data.Maybe ( isJust )
import qualified Data.Semigroup as S
+import GHC.Tc.Utils.Monad (getLclEnvLoc)
{-
************************************************************************
@@ -876,8 +877,8 @@ solveForAll :: CtEvidence -> [TyVar] -> TcThetaType -> PredType -> ExpansionFuel
solveForAll ev@(CtWanted { ctev_dest = dest, ctev_rewriters = rewriters, ctev_loc = loc })
tvs theta pred _fuel
= -- See Note [Solving a Wanted forall-constraint]
- setLclEnv (ctLocEnv loc) $
- -- This setLclEnv is important: the emitImplicationTcS uses that
+ setSrcSpan (getLclEnvLoc $ ctLocEnv loc) $
+ -- This setSrcSpan is important: the emitImplicationTcS uses that
-- TcLclEnv for the implication, and that in turn sets the location
-- for the Givens when solving the constraint (#21006)
do { let empty_subst = mkEmptySubst $ mkInScopeSet $
=====================================
compiler/GHC/Tc/Solver/Monad.hs
=====================================
@@ -57,7 +57,7 @@ module GHC.Tc.Solver.Monad (
getSolvedDicts, setSolvedDicts,
getInstEnvs, getFamInstEnvs, -- Getting the environments
- getTopEnv, getGblEnv, getLclEnv, setLclEnv,
+ getTopEnv, getGblEnv, getLclEnv, setSrcSpan,
getTcEvBindsVar, getTcLevel,
getTcEvTyCoVars, getTcEvBindsMap, setTcEvBindsMap,
tcLookupClass, tcLookupId, tcLookupTyCon,
@@ -194,6 +194,7 @@ import Data.IORef
import Data.List ( mapAccumL )
import Data.Foldable
import qualified Data.Semigroup as S
+import GHC.Types.SrcLoc
#if defined(DEBUG)
import GHC.Types.Unique.Set (nonDetEltsUniqSet)
@@ -1398,8 +1399,8 @@ getGblEnv = wrapTcS $ TcM.getGblEnv
getLclEnv :: TcS TcLclEnv
getLclEnv = wrapTcS $ TcM.getLclEnv
-setLclEnv :: TcLclEnv -> TcS a -> TcS a
-setLclEnv env = wrap2TcS (TcM.setLclEnv env)
+setSrcSpan :: RealSrcSpan -> TcS a -> TcS a
+setSrcSpan ss = wrap2TcS (TcM.setSrcSpan (RealSrcSpan ss mempty))
tcLookupClass :: Name -> TcS Class
tcLookupClass c = wrapTcS $ TcM.tcLookupClass c
=====================================
docs/users_guide/ghci.rst
=====================================
@@ -3546,41 +3546,19 @@ The interpreter can't load modules with foreign export declarations!
Unfortunately not. We haven't implemented it yet. Please compile any
offending modules by hand before loading them into GHCi.
-:ghc-flag:`-O` doesn't work with GHCi!
+:ghc-flag:`-O` is ineffective in GHCi!
.. index::
single: optimization; and GHCi
- For technical reasons, the bytecode compiler doesn't interact well
- with one of the optimisation passes, so we have disabled
- optimisation when using the interpreter. This isn't a great loss:
- you'll get a much bigger win by compiling the bits of your code that
- need to go fast, rather than interpreting them with optimisation
- turned on.
+ Before GHC 9.8, optimizations were considered too unstable to be used with
+ the bytecode interpreter.
+ This restriction has been lifted, but is still regarded as experimental and
+ guarded by :ghc-flag:`-funoptimized-core-for-interpreter`, which is enabled
+ by default.
+ In order to use optimizations, run: ::
-Modules using unboxed tuples or sums will automatically enable :ghc-flag:`-fobject-code`
-
- .. index::
- single: unboxed tuples, sums; and GHCi
-
- The bytecode interpreter doesn't support most uses of unboxed tuples or
- sums, so GHCi will automatically compile these modules, and all modules
- they depend on, to object code instead of bytecode.
-
- GHCi checks for the presence of unboxed tuples and sums in a somewhat
- conservative fashion: it simply checks to see if a module enables the
- :extension:`UnboxedTuples` or :extension:`UnboxedSums` language extensions.
- It is not always the case that code which enables :extension:`UnboxedTuples`
- or :extension:`UnboxedSums` requires :ghc-flag:`-fobject-code`, so if you
- *really* want to compile
- :extension:`UnboxedTuples`/:extension:`UnboxedSums`-using code to
- bytecode, you can do so explicitly by enabling the :ghc-flag:`-fbyte-code`
- flag. If you do this, do note that bytecode interpreter will throw an error
- if it encounters unboxed tuple/sum–related code that it cannot handle.
-
- Incidentally, the previous point, that :ghc-flag:`-O` is
- incompatible with GHCi, is because the bytecode compiler can't
- deal with unboxed tuples or sums.
+ ghci -fno-unoptimized-core-for-interpreter -O
Concurrent threads don't carry on running when GHCi is waiting for input.
This should work, as long as your GHCi was built with the
=====================================
libraries/base/GHC/Foreign.hs
=====================================
@@ -21,312 +21,22 @@ module GHC.Foreign (
-- * C strings with a configurable encoding
CString, CStringLen,
- -- conversion of C strings into Haskell strings
- --
+ -- * Conversion of C strings into Haskell strings
peekCString,
peekCStringLen,
- -- conversion of Haskell strings into C strings
- --
+ -- * Conversion of Haskell strings into C strings
newCString,
newCStringLen,
+ newCStringLen0,
- -- conversion of Haskell strings into C strings using temporary storage
- --
+ -- * Conversion of Haskell strings into C strings using temporary storage
withCString,
withCStringLen,
+ withCStringLen0,
withCStringsLen,
charIsRepresentable,
) where
-import Foreign.Marshal.Array
-import Foreign.C.Types
-import Foreign.Ptr
-import Foreign.Storable
-
-import Data.Word
-
--- Imports for the locale-encoding version of marshallers
-
-import Data.Tuple (fst)
-
-import GHC.Show ( show )
-
-import Foreign.Marshal.Alloc
-import Foreign.ForeignPtr
-
-import GHC.Debug
-import GHC.List
-import GHC.Num
-import GHC.Base
-
-import GHC.IO
-import GHC.IO.Exception
-import GHC.IO.Buffer
-import GHC.IO.Encoding.Types
-
-
-c_DEBUG_DUMP :: Bool
-c_DEBUG_DUMP = False
-
-putDebugMsg :: String -> IO ()
-putDebugMsg | c_DEBUG_DUMP = debugLn
- | otherwise = const (return ())
-
-
--- | A C string is a reference to an array of C characters terminated by NUL.
-type CString = Ptr CChar
-
--- | A string with explicit length information in bytes instead of a
--- terminating NUL (allowing NUL characters in the middle of the string).
-type CStringLen = (Ptr CChar, Int)
-
--- exported functions
--- ------------------
-
--- | Marshal a NUL terminated C string into a Haskell string.
---
-peekCString :: TextEncoding -> CString -> IO String
-peekCString enc cp = do
- sz <- lengthArray0 nUL cp
- peekEncodedCString enc (cp, sz * cCharSize)
-
--- | Marshal a C string with explicit length into a Haskell string.
---
-peekCStringLen :: TextEncoding -> CStringLen -> IO String
-peekCStringLen = peekEncodedCString
-
--- | Marshal a Haskell string into a NUL terminated C string.
---
--- * the Haskell string may /not/ contain any NUL characters
---
--- * new storage is allocated for the C string and must be
--- explicitly freed using 'Foreign.Marshal.Alloc.free' or
--- 'Foreign.Marshal.Alloc.finalizerFree'.
---
-newCString :: TextEncoding -> String -> IO CString
-newCString enc = liftM fst . newEncodedCString enc True
-
--- | Marshal a Haskell string into a C string (ie, character array) with
--- explicit length information.
---
--- * new storage is allocated for the C string and must be
--- explicitly freed using 'Foreign.Marshal.Alloc.free' or
--- 'Foreign.Marshal.Alloc.finalizerFree'.
---
-newCStringLen :: TextEncoding -> String -> IO CStringLen
-newCStringLen enc = newEncodedCString enc False
-
--- | Marshal a Haskell string into a NUL terminated C string using temporary
--- storage.
---
--- * the Haskell string may /not/ contain any NUL characters
---
--- * the memory is freed when the subcomputation terminates (either
--- normally or via an exception), so the pointer to the temporary
--- storage must /not/ be used after this.
---
-withCString :: TextEncoding -> String -> (CString -> IO a) -> IO a
-withCString enc s act = withEncodedCString enc True s $ \(cp, _sz) -> act cp
-
--- | Marshal a Haskell string into a C string (ie, character array)
--- in temporary storage, with explicit length information.
---
--- * the memory is freed when the subcomputation terminates (either
--- normally or via an exception), so the pointer to the temporary
--- storage must /not/ be used after this.
---
-withCStringLen :: TextEncoding -> String -> (CStringLen -> IO a) -> IO a
-withCStringLen enc = withEncodedCString enc False
-
--- | Marshal a list of Haskell strings into an array of NUL terminated C strings
--- using temporary storage.
---
--- * the Haskell strings may /not/ contain any NUL characters
---
--- * the memory is freed when the subcomputation terminates (either
--- normally or via an exception), so the pointer to the temporary
--- storage must /not/ be used after this.
---
-withCStringsLen :: TextEncoding
- -> [String]
- -> (Int -> Ptr CString -> IO a)
- -> IO a
-withCStringsLen enc strs f = go [] strs
- where
- go cs (s:ss) = withCString enc s $ \c -> go (c:cs) ss
- go cs [] = withArrayLen (reverse cs) f
-
--- | Determines whether a character can be accurately encoded in a
--- 'Foreign.C.String.CString'.
---
--- Pretty much anyone who uses this function is in a state of sin because
--- whether or not a character is encodable will, in general, depend on the
--- context in which it occurs.
-charIsRepresentable :: TextEncoding -> Char -> IO Bool
--- We force enc explicitly because `catch` is lazy in its
--- first argument. We would probably like to force c as well,
--- but unfortunately worker/wrapper produces very bad code for
--- that.
---
--- TODO If this function is performance-critical, it would probably
--- pay to use a single-character specialization of withCString. That
--- would allow worker/wrapper to actually eliminate Char boxes, and
--- would also get rid of the completely unnecessary cons allocation.
-charIsRepresentable !enc c =
- withCString enc [c]
- (\cstr -> do str <- peekCString enc cstr
- case str of
- [ch] | ch == c -> pure True
- _ -> pure False)
- `catch`
- \(_ :: IOException) -> pure False
-
--- auxiliary definitions
--- ----------------------
-
--- C's end of string character
-nUL :: CChar
-nUL = 0
-
--- Size of a CChar in bytes
-cCharSize :: Int
-cCharSize = sizeOf (undefined :: CChar)
-
-
-{-# INLINE peekEncodedCString #-}
-peekEncodedCString :: TextEncoding -- ^ Encoding of CString
- -> CStringLen
- -> IO String -- ^ String in Haskell terms
-peekEncodedCString (TextEncoding { mkTextDecoder = mk_decoder }) (p, sz_bytes)
- = bracket mk_decoder close $ \decoder -> do
- let chunk_size = sz_bytes `max` 1 -- Decode buffer chunk size in characters: one iteration only for ASCII
- !from0 <- fmap (\fp -> bufferAdd sz_bytes (emptyBuffer fp sz_bytes ReadBuffer)) $ newForeignPtr_ (castPtr p)
- !to <- newCharBuffer chunk_size WriteBuffer
-
- let go !iteration !from = do
- (why, from', !to') <- encode decoder from to
- if isEmptyBuffer from'
- then
- -- No input remaining: @why@ will be InputUnderflow, but we don't care
- withBuffer to' $ peekArray (bufferElems to')
- else do
- -- Input remaining: what went wrong?
- putDebugMsg ("peekEncodedCString: " ++ show iteration ++ " " ++ show why)
- (from'', to'') <- case why of InvalidSequence -> recover decoder from' to' -- These conditions are equally bad because
- InputUnderflow -> recover decoder from' to' -- they indicate malformed/truncated input
- OutputUnderflow -> return (from', to') -- We will have more space next time round
- putDebugMsg ("peekEncodedCString: from " ++ summaryBuffer from ++ " " ++ summaryBuffer from' ++ " " ++ summaryBuffer from'')
- putDebugMsg ("peekEncodedCString: to " ++ summaryBuffer to ++ " " ++ summaryBuffer to' ++ " " ++ summaryBuffer to'')
- to_chars <- withBuffer to'' $ peekArray (bufferElems to'')
- fmap (to_chars++) $ go (iteration + 1) from''
-
- go (0 :: Int) from0
-
-{-# INLINE withEncodedCString #-}
-withEncodedCString :: TextEncoding -- ^ Encoding of CString to create
- -> Bool -- ^ Null-terminate?
- -> String -- ^ String to encode
- -> (CStringLen -> IO a) -- ^ Worker that can safely use the allocated memory
- -> IO a
-withEncodedCString (TextEncoding { mkTextEncoder = mk_encoder }) null_terminate s act
- = bracket mk_encoder close $ \encoder -> withArrayLen s $ \sz p -> do
- from <- fmap (\fp -> bufferAdd sz (emptyBuffer fp sz ReadBuffer)) $ newForeignPtr_ p
-
- let go !iteration to_sz_bytes = do
- putDebugMsg ("withEncodedCString: " ++ show iteration)
- allocaBytes to_sz_bytes $ \to_p -> do
- -- See Note [Check *before* fill in withEncodedCString] about why
- -- this is subtle.
- mb_res <- tryFillBuffer encoder null_terminate from to_p to_sz_bytes
- case mb_res of
- Nothing -> go (iteration + 1) (to_sz_bytes * 2)
- Just to_buf -> withCStringBuffer to_buf null_terminate act
-
- -- If the input string is ASCII, this value will ensure we only allocate once
- go (0 :: Int) (cCharSize * (sz + 1))
-
-withCStringBuffer :: Buffer Word8 -> Bool -> (CStringLen -> IO r) -> IO r
-withCStringBuffer to_buf null_terminate act = do
- let bytes = bufferElems to_buf
- withBuffer to_buf $ \to_ptr -> do
- when null_terminate $ pokeElemOff to_ptr (bufR to_buf) 0
- act (castPtr to_ptr, bytes) -- NB: the length information is specified as being in *bytes*
-
-{-# INLINE newEncodedCString #-}
-newEncodedCString :: TextEncoding -- ^ Encoding of CString to create
- -> Bool -- ^ Null-terminate?
- -> String -- ^ String to encode
- -> IO CStringLen
-newEncodedCString (TextEncoding { mkTextEncoder = mk_encoder }) null_terminate s
- = bracket mk_encoder close $ \encoder -> withArrayLen s $ \sz p -> do
- from <- fmap (\fp -> bufferAdd sz (emptyBuffer fp sz ReadBuffer)) $ newForeignPtr_ p
-
- let go !iteration to_p to_sz_bytes = do
- putDebugMsg ("newEncodedCString: " ++ show iteration)
- mb_res <- tryFillBuffer encoder null_terminate from to_p to_sz_bytes
- case mb_res of
- Nothing -> do
- let to_sz_bytes' = to_sz_bytes * 2
- to_p' <- reallocBytes to_p to_sz_bytes'
- go (iteration + 1) to_p' to_sz_bytes'
- Just to_buf -> withCStringBuffer to_buf null_terminate return
-
- -- If the input string is ASCII, this value will ensure we only allocate once
- let to_sz_bytes = cCharSize * (sz + 1)
- to_p <- mallocBytes to_sz_bytes
- go (0 :: Int) to_p to_sz_bytes
-
-
-tryFillBuffer :: TextEncoder dstate -> Bool -> Buffer Char -> Ptr Word8 -> Int
- -> IO (Maybe (Buffer Word8))
-tryFillBuffer encoder null_terminate from0 to_p !to_sz_bytes = do
- !to_fp <- newForeignPtr_ to_p
- go (0 :: Int) from0 (emptyBuffer to_fp to_sz_bytes WriteBuffer)
- where
- go !iteration !from !to = do
- (why, from', to') <- encode encoder from to
- putDebugMsg ("tryFillBufferAndCall: " ++ show iteration ++ " " ++ show why ++ " " ++ summaryBuffer from ++ " " ++ summaryBuffer from')
- if isEmptyBuffer from'
- then if null_terminate && bufferAvailable to' == 0
- then return Nothing -- We had enough for the string but not the terminator: ask the caller for more buffer
- else return (Just to')
- else case why of -- We didn't consume all of the input
- InputUnderflow -> recover encoder from' to' >>= \(a,b) -> go (iteration + 1) a b -- These conditions are equally bad
- InvalidSequence -> recover encoder from' to' >>= \(a,b) -> go (iteration + 1) a b -- since the input was truncated/invalid
- OutputUnderflow -> return Nothing -- Oops, out of buffer during decoding: ask the caller for more
-{-
-Note [Check *before* fill in withEncodedCString]
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-It's very important that the size check and readjustment peformed by tryFillBuffer
-happens before the continuation is called. The size check is the part which can
-fail, the call to the continuation never fails and so the caller should respond
-first to the size check failing and *then* call the continuation. Making this evident
-to the compiler avoids historic space leaks.
-
-In a previous iteration of this code we had a pattern that, somewhat simplified,
-looked like this:
-
-go :: State -> (State -> IO a) -> IO a
-go state action =
- case tryFillBufferAndCall state action of
- Left state' -> go state' action
- Right result -> result
-
-`tryFillBufferAndCall` performed some checks, and then we either called action,
-or we modified the state and tried again.
-This went wrong because `action` can be a function closure containing a reference to
-a lazy data structure. If we call action directly, without retaining any references
-to action, that is fine. The data structure is consumed as it is produced and we operate
-in constant space.
-
-However the failure branch `go state' action` *does* capture a reference to action.
-This went wrong because the reference to action in the failure branch only becomes
-unreachable *after* action returns. This means we keep alive the function closure
-for `action` until `action` returns. Which in turn keeps alive the *whole* lazy list
-via `action` until the action has fully run.
-This went wrong in #20107, where the continuation kept an entire lazy bytestring alive
-rather than allowing it to be incrementally consumed and collected.
--}
-
+import GHC.Foreign.Internal
=====================================
libraries/base/GHC/Foreign/Internal.hs
=====================================
@@ -0,0 +1,357 @@
+{-# LANGUAGE Trustworthy #-}
+{-# LANGUAGE NoImplicitPrelude #-}
+{-# LANGUAGE ScopedTypeVariables #-}
+{-# LANGUAGE BangPatterns #-}
+
+-----------------------------------------------------------------------------
+-- |
+-- Module : GHC.Foreign.Internal
+-- Copyright : (c) The University of Glasgow, 2008-2011
+-- License : see libraries/base/LICENSE
+--
+-- Maintainer : libraries at haskell.org
+-- Stability : internal
+-- Portability : non-portable
+--
+-- Foreign marshalling support for CStrings with configurable encodings
+--
+-----------------------------------------------------------------------------
+
+module GHC.Foreign.Internal (
+ -- * C strings with a configurable encoding
+ CString, CStringLen,
+
+ -- * Conversion of C strings into Haskell strings
+ peekCString,
+ peekCStringLen,
+
+ -- * Conversion of Haskell strings into C strings
+ newCString,
+ newCStringLen,
+ newCStringLen0,
+
+ -- * Conversion of Haskell strings into C strings using temporary storage
+ withCString,
+ withCStringLen,
+ withCStringLen0,
+ withCStringsLen,
+
+ charIsRepresentable,
+ ) where
+
+import Foreign.Marshal.Array
+import Foreign.C.Types
+import Foreign.Ptr
+import Foreign.Storable
+
+import Data.Word
+
+-- Imports for the locale-encoding version of marshallers
+
+import Data.Tuple (fst)
+
+import GHC.Show ( show )
+
+import Foreign.Marshal.Alloc
+import Foreign.ForeignPtr
+
+import GHC.Debug
+import GHC.List
+import GHC.Num
+import GHC.Base
+
+import GHC.IO
+import GHC.IO.Exception
+import GHC.IO.Buffer
+import GHC.IO.Encoding.Types
+
+
+c_DEBUG_DUMP :: Bool
+c_DEBUG_DUMP = False
+
+putDebugMsg :: String -> IO ()
+putDebugMsg | c_DEBUG_DUMP = debugLn
+ | otherwise = const (return ())
+
+
+-- | A C string is a reference to an array of C characters terminated by NUL.
+type CString = Ptr CChar
+
+-- | A string with explicit length information in bytes instead of a
+-- terminating NUL (allowing NUL characters in the middle of the string).
+type CStringLen = (Ptr CChar, Int)
+
+-- exported functions
+-- ------------------
+
+-- | Marshal a NUL terminated C string into a Haskell string.
+--
+peekCString :: TextEncoding -> CString -> IO String
+peekCString enc cp = do
+ sz <- lengthArray0 nUL cp
+ peekEncodedCString enc (cp, sz * cCharSize)
+
+-- | Marshal a C string with explicit length into a Haskell string.
+--
+peekCStringLen :: TextEncoding -> CStringLen -> IO String
+peekCStringLen = peekEncodedCString
+
+-- | Marshal a Haskell string into a NUL terminated C string.
+--
+-- * the Haskell string may /not/ contain any NUL characters
+--
+-- * new storage is allocated for the C string and must be
+-- explicitly freed using 'Foreign.Marshal.Alloc.free' or
+-- 'Foreign.Marshal.Alloc.finalizerFree'.
+--
+newCString :: TextEncoding -> String -> IO CString
+newCString enc = liftM fst . newEncodedCString enc True
+
+-- | Marshal a Haskell string into a C string (ie, character array) with
+-- explicit length information.
+--
+-- Note that this does not NUL terminate the resulting string.
+--
+-- * new storage is allocated for the C string and must be
+-- explicitly freed using 'Foreign.Marshal.Alloc.free' or
+-- 'Foreign.Marshal.Alloc.finalizerFree'.
+--
+newCStringLen :: TextEncoding -> String -> IO CStringLen
+newCStringLen enc = newEncodedCString enc False
+
+-- | Marshal a Haskell string into a NUL terminated C string using temporary
+-- storage.
+--
+-- * the Haskell string may /not/ contain any NUL characters
+--
+-- * the memory is freed when the subcomputation terminates (either
+-- normally or via an exception), so the pointer to the temporary
+-- storage must /not/ be used after this.
+--
+withCString :: TextEncoding -> String -> (CString -> IO a) -> IO a
+withCString enc s act = withEncodedCString enc True s $ \(cp, _sz) -> act cp
+
+-- | Marshal a Haskell string into a C string (ie, character array)
+-- in temporary storage, with explicit length information.
+--
+-- Note that this does not NUL terminate the resulting string.
+--
+-- * the memory is freed when the subcomputation terminates (either
+-- normally or via an exception), so the pointer to the temporary
+-- storage must /not/ be used after this.
+--
+withCStringLen :: TextEncoding -> String -> (CStringLen -> IO a) -> IO a
+withCStringLen enc = withEncodedCString enc False
+
+-- | Marshal a Haskell string into a NUL-terminated C string (ie, character array)
+-- with explicit length information.
+--
+-- * new storage is allocated for the C string and must be
+-- explicitly freed using 'Foreign.Marshal.Alloc.free' or
+-- 'Foreign.Marshal.Alloc.finalizerFree'.
+--
+-- @since 4.19.0.0
+newCStringLen0 :: TextEncoding -> String -> IO CStringLen
+newCStringLen0 enc = newEncodedCString enc True
+
+-- | Marshal a Haskell string into a NUL-terminated C string (ie, character array)
+-- in temporary storage, with explicit length information.
+--
+-- * the memory is freed when the subcomputation terminates (either
+-- normally or via an exception), so the pointer to the temporary
+-- storage must /not/ be used after this.
+--
+-- @since 4.19.0.0
+withCStringLen0 :: TextEncoding -> String -> (CStringLen -> IO a) -> IO a
+withCStringLen0 enc = withEncodedCString enc True
+
+-- | Marshal a list of Haskell strings into an array of NUL terminated C strings
+-- using temporary storage.
+--
+-- * the Haskell strings may /not/ contain any NUL characters
+--
+-- * the memory is freed when the subcomputation terminates (either
+-- normally or via an exception), so the pointer to the temporary
+-- storage must /not/ be used after this.
+--
+withCStringsLen :: TextEncoding
+ -> [String]
+ -> (Int -> Ptr CString -> IO a)
+ -> IO a
+withCStringsLen enc strs f = go [] strs
+ where
+ go cs (s:ss) = withCString enc s $ \c -> go (c:cs) ss
+ go cs [] = withArrayLen (reverse cs) f
+
+-- | Determines whether a character can be accurately encoded in a
+-- 'Foreign.C.String.CString'.
+--
+-- Pretty much anyone who uses this function is in a state of sin because
+-- whether or not a character is encodable will, in general, depend on the
+-- context in which it occurs.
+charIsRepresentable :: TextEncoding -> Char -> IO Bool
+-- We force enc explicitly because `catch` is lazy in its
+-- first argument. We would probably like to force c as well,
+-- but unfortunately worker/wrapper produces very bad code for
+-- that.
+--
+-- TODO If this function is performance-critical, it would probably
+-- pay to use a single-character specialization of withCString. That
+-- would allow worker/wrapper to actually eliminate Char boxes, and
+-- would also get rid of the completely unnecessary cons allocation.
+charIsRepresentable !enc c =
+ withCString enc [c]
+ (\cstr -> do str <- peekCString enc cstr
+ case str of
+ [ch] | ch == c -> pure True
+ _ -> pure False)
+ `catch`
+ \(_ :: IOException) -> pure False
+
+-- auxiliary definitions
+-- ----------------------
+
+-- C's end of string character
+nUL :: CChar
+nUL = 0
+
+-- Size of a CChar in bytes
+cCharSize :: Int
+cCharSize = sizeOf (undefined :: CChar)
+
+
+{-# INLINE peekEncodedCString #-}
+peekEncodedCString :: TextEncoding -- ^ Encoding of CString
+ -> CStringLen
+ -> IO String -- ^ String in Haskell terms
+peekEncodedCString (TextEncoding { mkTextDecoder = mk_decoder }) (p, sz_bytes)
+ = bracket mk_decoder close $ \decoder -> do
+ let chunk_size = sz_bytes `max` 1 -- Decode buffer chunk size in characters: one iteration only for ASCII
+ !from0 <- fmap (\fp -> bufferAdd sz_bytes (emptyBuffer fp sz_bytes ReadBuffer)) $ newForeignPtr_ (castPtr p)
+ !to <- newCharBuffer chunk_size WriteBuffer
+
+ let go !iteration !from = do
+ (why, from', !to') <- encode decoder from to
+ if isEmptyBuffer from'
+ then
+ -- No input remaining: @why@ will be InputUnderflow, but we don't care
+ withBuffer to' $ peekArray (bufferElems to')
+ else do
+ -- Input remaining: what went wrong?
+ putDebugMsg ("peekEncodedCString: " ++ show iteration ++ " " ++ show why)
+ (from'', to'') <- case why of InvalidSequence -> recover decoder from' to' -- These conditions are equally bad because
+ InputUnderflow -> recover decoder from' to' -- they indicate malformed/truncated input
+ OutputUnderflow -> return (from', to') -- We will have more space next time round
+ putDebugMsg ("peekEncodedCString: from " ++ summaryBuffer from ++ " " ++ summaryBuffer from' ++ " " ++ summaryBuffer from'')
+ putDebugMsg ("peekEncodedCString: to " ++ summaryBuffer to ++ " " ++ summaryBuffer to' ++ " " ++ summaryBuffer to'')
+ to_chars <- withBuffer to'' $ peekArray (bufferElems to'')
+ fmap (to_chars++) $ go (iteration + 1) from''
+
+ go (0 :: Int) from0
+
+{-# INLINE withEncodedCString #-}
+withEncodedCString :: TextEncoding -- ^ Encoding of CString to create
+ -> Bool -- ^ Null-terminate?
+ -> String -- ^ String to encode
+ -> (CStringLen -> IO a) -- ^ Worker that can safely use the allocated memory
+ -> IO a
+withEncodedCString (TextEncoding { mkTextEncoder = mk_encoder }) null_terminate s act
+ = bracket mk_encoder close $ \encoder -> withArrayLen s $ \sz p -> do
+ from <- fmap (\fp -> bufferAdd sz (emptyBuffer fp sz ReadBuffer)) $ newForeignPtr_ p
+
+ let go !iteration to_sz_bytes = do
+ putDebugMsg ("withEncodedCString: " ++ show iteration)
+ allocaBytes to_sz_bytes $ \to_p -> do
+ -- See Note [Check *before* fill in withEncodedCString] about why
+ -- this is subtle.
+ mb_res <- tryFillBuffer encoder null_terminate from to_p to_sz_bytes
+ case mb_res of
+ Nothing -> go (iteration + 1) (to_sz_bytes * 2)
+ Just to_buf -> withCStringBuffer to_buf null_terminate act
+
+ -- If the input string is ASCII, this value will ensure we only allocate once
+ go (0 :: Int) (cCharSize * (sz + 1))
+
+withCStringBuffer :: Buffer Word8 -> Bool -> (CStringLen -> IO r) -> IO r
+withCStringBuffer to_buf null_terminate act = do
+ let bytes = bufferElems to_buf
+ withBuffer to_buf $ \to_ptr -> do
+ when null_terminate $ pokeElemOff to_ptr (bufR to_buf) 0
+ act (castPtr to_ptr, bytes) -- NB: the length information is specified as being in *bytes*
+
+{-# INLINE newEncodedCString #-}
+newEncodedCString :: TextEncoding -- ^ Encoding of CString to create
+ -> Bool -- ^ Null-terminate?
+ -> String -- ^ String to encode
+ -> IO CStringLen
+newEncodedCString (TextEncoding { mkTextEncoder = mk_encoder }) null_terminate s
+ = bracket mk_encoder close $ \encoder -> withArrayLen s $ \sz p -> do
+ from <- fmap (\fp -> bufferAdd sz (emptyBuffer fp sz ReadBuffer)) $ newForeignPtr_ p
+
+ let go !iteration to_p to_sz_bytes = do
+ putDebugMsg ("newEncodedCString: " ++ show iteration)
+ mb_res <- tryFillBuffer encoder null_terminate from to_p to_sz_bytes
+ case mb_res of
+ Nothing -> do
+ let to_sz_bytes' = to_sz_bytes * 2
+ to_p' <- reallocBytes to_p to_sz_bytes'
+ go (iteration + 1) to_p' to_sz_bytes'
+ Just to_buf -> withCStringBuffer to_buf null_terminate return
+
+ -- If the input string is ASCII, this value will ensure we only allocate once
+ let to_sz_bytes = cCharSize * (sz + 1)
+ to_p <- mallocBytes to_sz_bytes
+ go (0 :: Int) to_p to_sz_bytes
+
+
+tryFillBuffer :: TextEncoder dstate -> Bool -> Buffer Char -> Ptr Word8 -> Int
+ -> IO (Maybe (Buffer Word8))
+tryFillBuffer encoder null_terminate from0 to_p !to_sz_bytes = do
+ !to_fp <- newForeignPtr_ to_p
+ go (0 :: Int) from0 (emptyBuffer to_fp to_sz_bytes WriteBuffer)
+ where
+ go !iteration !from !to = do
+ (why, from', to') <- encode encoder from to
+ putDebugMsg ("tryFillBufferAndCall: " ++ show iteration ++ " " ++ show why ++ " " ++ summaryBuffer from ++ " " ++ summaryBuffer from')
+ if isEmptyBuffer from'
+ then if null_terminate && bufferAvailable to' == 0
+ then return Nothing -- We had enough for the string but not the terminator: ask the caller for more buffer
+ else return (Just to')
+ else case why of -- We didn't consume all of the input
+ InputUnderflow -> recover encoder from' to' >>= \(a,b) -> go (iteration + 1) a b -- These conditions are equally bad
+ InvalidSequence -> recover encoder from' to' >>= \(a,b) -> go (iteration + 1) a b -- since the input was truncated/invalid
+ OutputUnderflow -> return Nothing -- Oops, out of buffer during decoding: ask the caller for more
+{-
+Note [Check *before* fill in withEncodedCString]
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It's very important that the size check and readjustment peformed by tryFillBuffer
+happens before the continuation is called. The size check is the part which can
+fail, the call to the continuation never fails and so the caller should respond
+first to the size check failing and *then* call the continuation. Making this evident
+to the compiler avoids historic space leaks.
+
+In a previous iteration of this code we had a pattern that, somewhat simplified,
+looked like this:
+
+go :: State -> (State -> IO a) -> IO a
+go state action =
+ case tryFillBufferAndCall state action of
+ Left state' -> go state' action
+ Right result -> result
+
+`tryFillBufferAndCall` performed some checks, and then we either called action,
+or we modified the state and tried again.
+This went wrong because `action` can be a function closure containing a reference to
+a lazy data structure. If we call action directly, without retaining any references
+to action, that is fine. The data structure is consumed as it is produced and we operate
+in constant space.
+
+However the failure branch `go state' action` *does* capture a reference to action.
+This went wrong because the reference to action in the failure branch only becomes
+unreachable *after* action returns. This means we keep alive the function closure
+for `action` until `action` returns. Which in turn keeps alive the *whole* lazy list
+via `action` until the action has fully run.
+This went wrong in #20107, where the continuation kept an entire lazy bytestring alive
+rather than allowing it to be incrementally consumed and collected.
+-}
+
=====================================
libraries/base/System/Posix/Internals.hs
=====================================
@@ -34,7 +34,6 @@ import System.Posix.Types
import Foreign
import Foreign.C
--- import Data.Bits
import Data.Maybe
#if !defined(HTYPE_TCFLAG_T)
@@ -51,6 +50,9 @@ import GHC.IO.Device
#if !defined(mingw32_HOST_OS)
import {-# SOURCE #-} GHC.IO.Encoding (getFileSystemEncoding)
import qualified GHC.Foreign as GHC
+import GHC.Ptr
+#else
+import Data.OldList (elem)
#endif
-- ---------------------------------------------------------------------------
@@ -164,13 +166,23 @@ fdGetMode fd = do
#if defined(mingw32_HOST_OS)
withFilePath :: FilePath -> (CWString -> IO a) -> IO a
-withFilePath = withCWString
+withFilePath fp f = do
+ checkForInteriorNuls fp
+ withCWString fp f
newFilePath :: FilePath -> IO CWString
-newFilePath = newCWString
+newFilePath fp = do
+ checkForInteriorNuls fp
+ newCWString fp
peekFilePath :: CWString -> IO FilePath
peekFilePath = peekCWString
+
+-- | Check a 'FilePath' for internal NUL codepoints as these are
+-- disallowed in Windows filepaths. See #13660.
+checkForInteriorNuls :: FilePath -> IO ()
+checkForInteriorNuls fp = when ('\0' `elem` fp) (throwInternalNulError fp)
+
#else
withFilePath :: FilePath -> (CString -> IO a) -> IO a
@@ -178,13 +190,43 @@ newFilePath :: FilePath -> IO CString
peekFilePath :: CString -> IO FilePath
peekFilePathLen :: CStringLen -> IO FilePath
-withFilePath fp f = getFileSystemEncoding >>= \enc -> GHC.withCString enc fp f
-newFilePath fp = getFileSystemEncoding >>= \enc -> GHC.newCString enc fp
+withFilePath fp f = do
+ enc <- getFileSystemEncoding
+ GHC.withCStringLen0 enc fp $ \(str, len) -> do
+ checkForInteriorNuls fp (str, len)
+ f str
+newFilePath fp = do
+ enc <- getFileSystemEncoding
+ (str, len) <- GHC.newCStringLen0 enc fp
+ checkForInteriorNuls fp (str, len)
+ return str
peekFilePath fp = getFileSystemEncoding >>= \enc -> GHC.peekCString enc fp
peekFilePathLen fp = getFileSystemEncoding >>= \enc -> GHC.peekCStringLen enc fp
+-- | Check an encoded 'FilePath' for internal NUL octets as these are
+-- disallowed in POSIX filepaths. See #13660.
+checkForInteriorNuls :: FilePath -> CStringLen -> IO ()
+checkForInteriorNuls fp (str, len) =
+ when (len' /= len) (throwInternalNulError fp)
+ -- N.B. If the string contains internal NUL codeunits then the strlen will
+ -- indicate a size smaller than that returned by withCStringLen.
+ where
+ len' = case str of Ptr ptr -> I# (cstringLength# ptr)
#endif
+throwInternalNulError :: FilePath -> IO a
+throwInternalNulError fp = ioError err
+ where
+ err =
+ IOError
+ { ioe_handle = Nothing
+ , ioe_type = InvalidArgument
+ , ioe_location = "checkForInteriorNuls"
+ , ioe_description = "FilePaths must not contain internal NUL code units."
+ , ioe_errno = Nothing
+ , ioe_filename = Just fp
+ }
+
-- ---------------------------------------------------------------------------
-- Terminal-related stuff
=====================================
libraries/base/base.cabal
=====================================
@@ -351,6 +351,7 @@ Library
GHC.Event.IntVar
GHC.Event.PSQ
GHC.Event.Unique
+ GHC.Foreign.Internal
-- GHC.IOPort -- TODO: hide again after debug
GHC.Unicode.Internal.Bits
GHC.Unicode.Internal.Char.DerivedCoreProperties
=====================================
libraries/base/tests/T13660.hs
=====================================
@@ -0,0 +1,11 @@
+-- | This should print an InvalidArgument error complaining that
+-- the file path contains a NUL octet.
+module Main where
+
+import System.IO.Error
+
+main :: IO ()
+main = do
+ catchIOError
+ (writeFile "hello\x00world" "hello")
+ print
=====================================
libraries/base/tests/T13660.stdout
=====================================
Binary files /dev/null and b/libraries/base/tests/T13660.stdout differ
=====================================
libraries/base/tests/all.T
=====================================
@@ -256,6 +256,7 @@ test('T13191',
['-O'])
test('T13525', [when(opsys('mingw32'), skip), js_broken(22374), req_process], compile_and_run, [''])
test('T13097', normal, compile_and_run, [''])
+test('T13660', when(opsys('mingw32'), skip), compile_and_run, [''])
test('functorOperators', normal, compile_and_run, [''])
test('T3474',
[collect_stats('max_bytes_used',5),
=====================================
testsuite/tests/ghc-api/T10052/T10052.stderr
=====================================
@@ -1,3 +1,3 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
=====================================
testsuite/tests/ghci.debugger/scripts/print007.stderr
=====================================
@@ -1,3 +1,3 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
=====================================
testsuite/tests/ghci/should_fail/T10549.stderr
=====================================
@@ -1,3 +1,3 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
=====================================
testsuite/tests/ghci/should_fail/T10549a.stderr
=====================================
@@ -1,3 +1,3 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
=====================================
testsuite/tests/safeHaskell/ghci/p14.stderr
=====================================
@@ -1,6 +1,6 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
<interactive>:10:25: error: [GHC-39999]
• No instance for ‘Num a’ arising from a use of ‘f’
=====================================
testsuite/tests/th/T8333.stderr
=====================================
@@ -1,3 +1,3 @@
when making flags consistent: warning:
- Optimization flags are incompatible with the byte-code interpreter; optimization flags ignored.
+ Using optimization flags with the byte-code interpreter is experimental. Pass -fno-unoptimized-core-for-interpreter to enable this feature.
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/14b690edc8c8883e04a7632a6e73b8aace099497...70c6bca275ee171ae8f5828a59cb810614e88c49
--
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/14b690edc8c8883e04a7632a6e73b8aace099497...70c6bca275ee171ae8f5828a59cb810614e88c49
You're receiving this email because of your account on gitlab.haskell.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20230518/27db15ea/attachment-0001.html>
More information about the ghc-commits
mailing list