Adding new contructors to IOMode to support "don't overwrite if already exists" behavior

Mon May 11 16:12:14 UTC 2020

O_NOFOLLOW|O_PATH is more or less the same as O_SYMLINK: it gives you
a handle on a symlink. But there are semantic differences including in
how you make use of it. O_NOFOLLOW by itself is useful for secure
applications; O_PATH more or less mandates system-specific code that
we tend to either not support or fold into POSIX or Win32 support.

The file locking flags are a minefield: they don't mean the same thing
on Windows vs. POSIX, and may have semantic differences between POSIX
platforms. It's better to use the low level calls as described above
if you need locking, since you can then write the appropriate code for
the platform and your use case.

O_DIRECTORY by itself isn't problematic, but only does something
useful on FreeBSD (you can read a directory's raw contents using the
dirent code). Other calls are system dependent as above, and force you
into POSIX if supported at all.

In short, there's good reason for most of the open flags to not be
supported in the OS-independent openFile. O_EXCL and maybe O_NOFOLLOW
are the most likely to be portably useful.

On 5/11/20, Victoria Mitchell <victoria at quietmisdreavus.net> wrote:
> Hi all!
>
> I took another look at this this past weekend and came up with a different
> solution, that might fit Sven's idea of a more-general mechanism to expose
> the `open()` flags.
>
> I have a new branch open on GitLab:
> https://gitlab.haskell.org/QuietMisdreavus/ghc/-/commit/ed8f142f9e18f8ef7389b8e8e4d722dcaf740971?w=1
> <https://gitlab.haskell.org/QuietMisdreavus/ghc/-/commit/ed8f142f9e18f8ef7389b8e8e4d722dcaf740971?view=parallel&w=1>
> I'm curious at what point i should turn this branch into an MR.
>
> The rough idea is that i introduced a new `IOFlags` type, which wraps the
> idea of `open()` flags, and added some new "openFileWithFlags" functions
> that take this new type in addition to the existing `IOMode`. These flags
> are then combined with the base ones from `IOMode` to create the final set
> of flags given to `open()`. I decided to keep all the existing "openFile"
> (et al) behavior w.r.t. always creating the file, clearing the file if
> `WriteMode`/`ReadWriteMode` were given instead of `AppendMode`, etc.
>
> In the initial commit, i only added EXCL since all the other existing flags
> are exposed by either existing IOMode behavior or in other functions, but it
> could be possible to add new flags.
>
> -----
>
> Speaking of additional flags to support, i did some additional research into
> the different sets of flags exposed on different platforms. I'm choosing to
> ignore the MSVC version of `open()` for the moment, since MSYS is used on
> Windows currently, and they expose the Linux version of the syscall.
>
> For example, here's an excerpt from the man page for `open(2)`, on my macOS
> system, talking about `oflags`:
>
> ```
>  The flags specified for the oflag argument must include exactly one of the
> following
>  file access modes:
>
>  O_RDONLY open for reading only
>  O_WRONLY open for writing only
>  O_RDWR open for reading and writing
>
>  In addition any combination of the following values can be or'ed in oflag:
>
>  O_NONBLOCK do not block on open or for data to become available
>  O_APPEND append on each write
>  O_CREAT create file if it does not exist
>  O_TRUNC truncate size to 0
>  O_EXCL error if O_CREAT and the file exists
>  O_SHLOCK atomically obtain a shared lock
>  O_EXLOCK atomically obtain an exclusive lock
>  O_NOFOLLOW do not follow symlinks
>  O_SYMLINK allow open of symlinks
>  O_EVTONLY descriptor requested for event notifications only
>  O_CLOEXEC mark as close-on-exec
> ```
>
> (I'm not sure if there are up-to-date man pages for macOS online any more?
> The only pages i could find on the Apple Developer site were out-of-date
> pages labeled for iOS, most recently updated in 2016.)
>
> Just from this list, it's worthwhile to winnow out the flags that are
> already exposed or made irrelevant by existing behavior. For example,
> O_CREAT is always passed for IOModes that open a file for writing, and
> O_APPEND is already exposed in AppendMode. O_TRUNC is done manually in
> GHC.IO.FD.openFile' when the IOMode of WriteMode is passed in. O_SHLOCK and
> O_EXLOCK are also done inside the runtime in C code in `rts/fileLock.c`.
> (Would it be worthwhile to expose these flags anyway to allow for
> system-level locking?) It looks like O_SYMLINK is macOS-exclusive, since it
> doesn't appear in the pages for Linux or FreeBSD (linked below). O_CLOEXEC
> sounds like behavior that's otherwise not exposed, but is it valid to
> `exec()` here? I'm admittedly not super familiar with all the internals that
> would need to be accounted for in a fork/exec combo.
>
> To my eyes, that leaves O_EXCL (which i've exposed in my commit), O_NONBLOCK
> (which is currently exposed in separate functions which i did not wrap in my
> commit), O_NOFOLLOW, and maybe O_SHLOCK/O_EXLOCK. This also doesn't mention
> Linux/BSD-exclusive flags that are not exposed on macOS.
>
> For reference, here is the Linux page for `open(3p)`:
> http://man7.org/linux/man-pages/man3/open.3p.html
> and the FreeBSD page for `open(2)`:
> https://www.freebsd.org/cgi/man.cgi?query=open&sektion=2&manpath=FreeBSD+12.1-RELEASE+and+Ports
>
> These mention even more flags, including the additional modes of O_EXEC and
> (on Linux) O_SEARCH. An interesting flag not mentioned in the macOS page is
> O_SYNC (and friends, O_FSYNC/O_DIRECT on BSD and O_DSYNC/O_RSYNC on Linux),
> which provide extra filesystem synchronization outside of the runtime. They
> both also mention O_DIRECTORY, which seems like a good guard to expose
> similarly to O_EXCL or O_NOFOLLOW.
>
> If necessary, we could expose the IOFlags constructor and allow libraries
> like `unix` to expose system-specific flags themselves.
>
> -----
>
> Regardless, i'm curious about everyone's impression of this approach. This
> should allow a more extensible method to expose these additional flags,
> which should sidestep the previous concerns about naming the new IOModes or
> creating "new ad-hoc combinations of flags".
>
> Thanks,
> Victoria Mitchell (formerly Grey Mitchell) (@QuietMisdreavus)
> https://quietmisdreavus.net/
>
> On Wed, Apr 1, 2020, at 12:45 AM, Sven Panne wrote:
>> Let's have a look at the POSIX spec for open():
>> https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html It
>> very clearly distinguishes between *2* disjoint sets of flags:
>>
>>  * You have to use exactly one of O_EXEC, O_RDONLY, O_RDWR, O_SEARCH, and
>> O_WRONLY.
>>
>>  * In addition, you can specify any combination of O_APPEND, O_CLOEXEC,
>> O_CREAT, O_DIRECTORY, O_DSYNC, O_EXCL, O_NOCTTY, O_NOFOLLOW, O_NONBLOCK,
>> O_RSYNC, O_SYNC, O_TRUNC, and O_TTY_INIT.
>>
>> Alas, GHC.IO.FD and GHC.IO.IOMode completely ignore this distinction
>> already and add tons of special cases:
>>
>>  * They add an ad hoc combination of O_WRONLY and O_APPEND (AppendMode).
>>
>>  * They leave out O_EXEC and O_SEARCH.
>>
>>  * They add an ad hoc boolean flag for non-blocking I/O to openFile, i.e.
>> pick 1 of the 13 possible additional flags.
>>
>>  * For some obscure reason, O_NOCTTY is always added.
>>
>>  * HandleType doesn't reflect the first set of flags.
>>
>> My proposal is to not make this mess even worse by adding yet another ad
>> hoc combination, but to think about how to expose these two sets in a sane
>> way.
>>
>> I didn't have a look at the Windows counterpart or into all the *nix
>> variants yet, but I guess that the overlap in the API is quite big. But
>> even the intersection of all the relevant platforms will very probably
>> have the distinction between the 2 sets of flags, so we should reflect
>> that in the Haskell API, too.
>

-- 
brandon s allbery kf8nh
allbery.b at gmail.com